How to Run a Local AI Chatbot on Your Server with Ollama and Open WebUI

You can run a capable language model entirely on your own server — nothing leaves the house, no subscription, no account. Ollama downloads and runs the models; Open WebUI gives you a chat page that looks like the commercial ones, with user accounts, document upload and chat history. The catch is hardware: a modern GPU makes it fast, a CPU makes it usable for short answers, and a Raspberry Pi makes it a demo.

A mini PC with 16 GB and no GPU runs a 7B model at a few words per second. That's enough for summaries, rewrites and questions; not enough for long documents.

Installing on the host rather than in Docker makes GPU access simpler:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama run llama3.2

Type a question; /bye to leave. If you have an NVIDIA card with drivers installed (nvidia-smi works), Ollama uses it automatically.

Let Docker containers reach it by making it listen on all interfaces:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

sudo systemctl restart ollama

mkdir -p ~/docker/open-webui && cd ~/docker/open-webui
nano compose.yaml

services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "3000:8080"
environment:
OLLAMA_BASE_URL: http://host.docker.internal:11434
WEBUI_AUTH: "true"
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- ./data:/app/backend/data
restart: unless-stopped

docker compose up -d
sudo ufw allow from 192.168.1.0/24 to any port 3000

Open http://192.168.1.50:3000, create the first account (it becomes admin). Pick the model at the top and chat.

ollama pull qwen2.5:7b # strong general model
ollama pull deepseek-r1:8b # reasoning
ollama pull llava # describes images
ollama pull nomic-embed-text # for document search in Open WebUI
ollama list
ollama rm llama3.2

Models are a few GB each; keep an eye on df -h.

In Open WebUI, Workspace → Knowledge → +, upload PDFs, then reference the collection with # in a chat. It pulls relevant passages into the model's context. Set the embedding model to nomic-embed-text under Admin → Settings → Documents for better results.

Admin → Users — add family accounts, or set Default user role to pending so new sign-ups wait for approval. Put it behind the reverse proxy and VPN rather than the open internet; a chatbot with your documents in it is private by definition.

NVIDIA: install the nvidia-container-toolkit, then add deploy: resources: reservations: devices: - driver: nvidia, count: all, capabilities: [gpu] to an ollama/ollama service. AMD: use ollama/ollama:rocm with /dev/kfd and /dev/dri devices. The host install avoids all of this.

Open WebUI shows no models. It can't reach Ollama. curl http://192.168.1.50:11434 from the server; check OLLAMA_HOST=0.0.0.0 and the firewall (sudo ufw allow from 172.16.0.0/12 to any port 11434 for Docker's networks).

Answers crawl. The model doesn't fit in RAM and is swapping, or the GPU isn't being used (ollama ps shows 100% CPU). Use a smaller model or quantisation (:q4_K_M tags).

"Error: model requires more system memory". Exactly that. Smaller model.

Server fans scream. Normal while generating. ollama ps shows models unload after five minutes idle; set OLLAMA_KEEP_ALIVE=0 to unload immediately.

Ollama on the host, one model that fits your RAM, Open WebUI in Docker, LAN-only access. A private assistant that reads your documents and never phones home — fast with a GPU, patient without one, and entirely under your roof.