Ollama on a Homelab: Local LLMs Without Melting the Rack

What Ollama is (operator view)

Ollama is a local runtime for open large language models. You pull a model, chat on the CLI, or call a REST API on port 11434. Upstream lives at github.com/ollama/ollama (MIT). The product site and model library are at ollama.com and ollama.com/library.

Think of it as “Docker-ish ergonomics for GGUF-style local inference,” not a full chat product. Chat UIs (Open WebUI, LibreChat, AnythingLLM) and coding agents sit on top. Orivana operator notes cover install paths and day-two ops here; upstream docs remain the source for API details.

Who this guide is for

Homelab operators who want private drafting, coding help, or RAG experiments without sending every prompt to a cloud vendor. If you need multi-tenant SaaS SLAs, stick with a hosted API. If you need “one box, one family, models stay on disk,” Ollama fits.

Install paths that stay boring

Official installers cover macOS, Windows, and Linux (`curl -fsSL https://ollama.com/install.sh | sh` on Linux/macOS; PowerShell installer on Windows). For a rack or NAS-adjacent Linux host, the official Docker image `ollama/ollama` on Docker Hub is usually the cleanest Compose citizen.

Prefer one install path per machine. Mixing a desktop app, a systemd unit, and a Compose service on the same host fights over ports and model directories.

Docker Compose sketch

A minimal service publishes only what you need. Bind `11434` to localhost or a Docker network — not `0.0.0.0` on a public VPS. Mount a named volume for models; they are the data gravity.

services:
  ollama:
    image: ollama/ollama:0.9.6
    volumes:
      - ollama_models:/root/.ollama
    # ports:
    #   - "127.0.0.1:11434:11434"
    restart: unless-stopped
volumes:
  ollama_models:

Pin the tag you tested. After pull, record the digest in STACK.md the same way you would for Immich or Vaultwarden:

    image: ollama/ollama:0.9.6@sha256:<digest-from-docker-inspect>

Add NVIDIA Container Toolkit (or your vendor’s equivalent) only after CPU-only chat works. GPU pass-through failures are easier to debug when the API already answers.

First commands after install

Pull a small model before a 70B fantasy: `ollama pull` a lightweight tag from the library, then `ollama run <model>`. Confirm `curl http://127.0.0.1:11434/api/tags` lists what you expect.

Chat API shape (see docs.ollama.com/api):

curl http://127.0.0.1:11434/api/chat -d '{
  "model": "gemma3",
  "messages": [{ "role": "user", "content": "Say hello in one sentence." }],
  "stream": false
}'

Official SDKs: ollama-python and ollama-js.

Disk, RAM, and model gravity

Models are multi-gigabyte artifacts. Plan SSD space like you plan Immich libraries: a volume you can back up, prune, and measure. `ollama list` shows what is on disk; unused tags accumulate quietly.

When disk-full alerts hit during unrelated Immich imports, the culprit was often Ollama experiments, not photos. Prune on a schedule:

ollama list
ollama rm llama3.2:70b-instruct-q4_0   # example — remove what you no longer need
docker system df -v   # confirm named volume shrink expectations

Inside Docker, the models live in the named volume (`ollama_models` above). Back up that volume with restic or snapshot the Proxmox disk — re-downloading a 40GB model over residential uplink is not a restore strategy.

RAM/VRAM rules of thumb: if the model does not fit comfortably, expect swap thrash or refusal — not magic. Start small; grow only after the backup and update ritual exists.

GPU toolkit: CPU first, then pass-through

Lab note: we tried NVIDIA Container Toolkit on a fresh Proxmox VM before confirming CPU inference. Compose failed with obscure `nvidia-container-cli` errors; debugging consumed an evening.

Working order: (1) CPU-only compose, API responds, small model chats. (2) Install toolkit on the host per NVIDIA docs. (3) Add `deploy.resources.reservations.devices` or runtime flags to compose. (4) One model pull, one chat. If GPU fails, you still have a working CPU box — not a brick.

Splitting "experiment box" (CPU mini-PC) from "family UI box" (GPU host) reduced complaints when the GPU path was mid-debug.

Exposure and threat model

An open Ollama port on the public internet is an unauthenticated inference endpoint. Treat it like an unauthenticated database admin UI. Keep it on localhost, a private Docker network, or VPN.

**Failure story:** we published `11434:11434` on a cheap VPS "for a quick remote test." Shodan-style scanners hit it within hours — same lesson as open Postgres or Redis. Logs showed foreign IPs probing `/api/chat`. Fix: firewall deny, bind to localhost, access via WireGuard only.

If a LAN UI needs access, put Open WebUI (or similar) behind your reverse proxy with auth — do not publish raw `:11434`.

Prompts and embeddings can contain secrets. Local inference reduces vendor logging; it does not erase disk forensics. Encrypt volumes if the box leaves the house.

Pairing a UI without boiling the ocean

Weekend one: Ollama only, CLI + API health. Weekend two: one UI (Open WebUI is a common choice) on an internal network, TLS at Caddy/Traefik, no public Ollama port. Coding agent integrations (`ollama launch …` per upstream docs) come after the API is stable.

Updates without fear

Read release notes before `pull`. Snapshot the models volume if you care about a specific digest set. After upgrade: `api/tags`, one chat completion, then the UI login path.

Rollback is previous image digest + same volume — edit compose, `docker compose down && docker compose up -d`. Models usually survive across runtime upgrades; still verify one tagged model loads.

Lab notes from careful ops practice

We keep Ollama on an internal Compose network; only the chat UI sits behind the reverse proxy.

Publishing 11434 "temporarily" on a VPS attracted scanners within hours — firewall and VPN-only access are non-negotiable for any host with a public IP.

Model volume grew faster than photo metadata DBs. A monthly `ollama list` prune stopped surprise disk-full alerts during unrelated Immich imports.

CPU-only chat on a small mini-PC was fine for drafting; anything interactive for the household waited on a GPU host.

Pinning `ollama/ollama` by digest after the first successful weekend prevented a surprise `:latest` pull from breaking Open WebUI's expected API shape.

When not to self-host Ollama

Skip it if you cannot spare disk or quiet hours for updates, if every user needs global low-latency, or if compliance requires a vendor DPA you already have with a cloud LLM. Hybrid is fine: local for private notes, cloud for burst coding.

Related reading

See Docker Compose hardening, Updating without fear, Secure remote access, Cost of self-hosting, and the Homelab ops essentials collection for adjacent operator notes.

Upstream links

• Source: https://github.com/ollama/ollama
• Docs: https://docs.ollama.com
• Library: https://ollama.com/library