Ollama on a Homelab: Local LLMs Without Melting the Rack

What Ollama is (operator view)

Ollama is a local runtime for open large language models. You pull a model, chat on the CLI, or call a REST API on port 11434. Upstream lives at github.com/ollama/ollama (MIT). The product site and model library are at ollama.com and ollama.com/library.

Think of it as “Docker-ish ergonomics for GGUF-style local inference,” not a full chat product. Chat UIs (Open WebUI, LibreChat, AnythingLLM) and coding agents sit on top. On Orivana the catalog entry is Ollama.

Who this guide is for

Homelab operators who want private drafting, coding help, or RAG experiments without sending every prompt to a cloud vendor. If you need multi-tenant SaaS SLAs, stick with a hosted API. If you need “one box, one family, models stay on disk,” Ollama fits.

Install paths that stay boring

Official installers cover macOS, Windows, and Linux (`curl -fsSL https://ollama.com/install.sh | sh` on Linux/macOS; PowerShell installer on Windows). For a rack or NAS-adjacent Linux host, the official Docker image `ollama/ollama` on Docker Hub is usually the cleanest Compose citizen.

Prefer one install path per machine. Mixing a desktop app, a systemd unit, and a Compose service on the same host fights over ports and model directories.

Docker Compose sketch

A minimal service publishes only what you need. Bind `11434` to localhost or a Docker network — not `0.0.0.0` on a public VPS. Mount a named volume for models; they are the data gravity.

```yaml services: ollama: image: ollama/ollama:latest volumes: - ollama_models:/root/.ollama # ports: # - "127.0.0.1:11434:11434" restart: unless-stopped volumes: ollama_models: ```

Add NVIDIA Container Toolkit (or your vendor’s equivalent) only after CPU-only chat works. GPU pass-through failures are easier to debug when the API already answers.

First commands after install

Pull a small model before a 70B fantasy: `ollama pull` a lightweight tag from the library, then `ollama run <model>`. Confirm `curl http://127.0.0.1:11434/api/tags` lists what you expect.

Chat API shape (see docs.ollama.com/api):

```bash curl http://127.0.0.1:11434/api/chat -d '{ "model": "gemma3", "messages": [{ "role": "user", "content": "Say hello in one sentence." }], "stream": false }' ```

Official SDKs: ollama-python and ollama-js.

Disk, RAM, and model gravity

Models are multi-gigabyte artifacts. Plan SSD space like you plan Immich libraries: a volume you can back up, prune, and measure. `ollama list` / delete unused tags on a schedule or the disk fills during “just one more experiment.”

RAM/VRAM rules of thumb: if the model does not fit comfortably, expect swap thrash or refusal — not magic. Start small; grow only after the backup and update ritual exists.

Exposure and threat model

An open Ollama port on the public internet is an unauthenticated inference endpoint. Treat it like an unauthenticated database admin UI. Keep it on localhost, a private Docker network, or VPN. If a LAN UI needs access, put Open WebUI (or similar) behind your reverse proxy with auth — do not publish raw `:11434`.

Prompts and embeddings can contain secrets. Local inference reduces vendor logging; it does not erase disk forensics. Encrypt volumes if the box leaves the house.

Pairing a UI without boiling the ocean

Weekend one: Ollama only, CLI + API health. Weekend two: one UI (Open WebUI is a common choice) on an internal network, TLS at Caddy/Traefik, no public Ollama port. Coding agent integrations (`ollama launch …` per upstream docs) come after the API is stable.

Updates without fear

Pin the image tag you tested. Read release notes before `pull`. Snapshot the models volume if you care about a specific digest set. After upgrade: `api/tags`, one chat completion, then the UI login path. Rollback is “previous image tag + same volume,” not “re-download the internet.”

Lab notes from Orivana benches

We keep Ollama on an internal Compose network; only the chat UI sits behind the reverse proxy. Publishing 11434 “temporarily” on a VPS attracted scanners within hours — same lesson as open Postgres.

Model volume grew faster than photo metadata DBs. A monthly `ollama list` prune stopped surprise disk-full alerts during unrelated Immich imports.

CPU-only chat on a small mini-PC was fine for drafting; anything interactive for the household waited on a GPU host. Split “experiment box” vs “family UI box” reduced complaints.

When not to self-host Ollama

Skip it if you cannot spare disk or quiet hours for updates, if every user needs global low-latency, or if compliance requires a vendor DPA you already have with a cloud LLM. Hybrid is fine: local for private notes, cloud for burst coding.

Related reading

See Docker Compose hardening, Updating without fear, Secure remote access, Cost of self-hosting, and the catalog page for Ollama.

Upstream links

- Source: https://github.com/ollama/ollama

- Docs: https://docs.ollama.com

- Library: https://ollama.com/library