How to Collect All Your Logs with Loki and Grafana (Search Months of Logs from Every Container and Host)

Dozzle (125) is the live view. The journal (14) holds the host's logs. Docker's logs rotate away in days (67). When "something odd happened last week" needs an answer, you want every log from every container and every machine in one searchable place, kept for a month. Loki is Grafana's log store — it indexes labels, not text, so it's light enough for a home server — and Alloy (Grafana's collector; Promtail is its predecessor) ships logs into it. You query them in the Grafana you already run (90).

In ~/docker/monitoring/compose.yaml (tutorial 90), add:

loki:
image: grafana/loki:latest
container_name: loki
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki-config.yaml:/etc/loki/local-config.yaml
- ./loki-data:/loki
restart: unless-stopped

alloy:
image: grafana/alloy:latest
container_name: alloy
command: run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
volumes:
- ./config.alloy:/etc/alloy/config.alloy
- /var/run/docker.sock:/var/run/docker.sock:ro
- /var/log/journal:/var/log/journal:ro
- /run/log/journal:/run/log/journal:ro
- /etc/machine-id:/etc/machine-id:ro
restart: unless-stopped

Neither needs a published port; Grafana reaches Loki by name.

nano loki-config.yaml

auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 720h
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystem

720h = 30 days. mkdir loki-data && sudo chown 10001:10001 loki-data.

nano config.alloy

// Docker containers: every container's stdout/stderr, labelled by name
discovery.docker "containers" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "containers" {
targets = discovery.docker.containers.targets
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/(.*)"
target_label = "container"
}
}
loki.source.docker "containers" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.containers.output
forward_to = [loki.write.local.receiver]
labels = { job = "docker", host = "homeserver" }
}

// The host's journal: sshd, cron, kernel, everything
loki.source.journal "host" {
forward_to = [loki.write.local.receiver]
labels = { job = "journal", host = "homeserver" }
relabel_rules = loki.relabel.journal.rules
}
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}

loki.write "local" {
endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

docker compose up -d
docker compose logs alloy | tail

Connections → Data sources → Add → Loki, URL http://loki:3100, Save & test. Then Explore, pick Loki, and the label browser shows container, unit, host. Queries (LogQL):

{container="nginx"} # one container's log
{container="nginx"} |= "error" # lines containing error
{job="journal", unit="ssh.service"} |= "Failed" # SSH failures from the host journal
{job=~".+"} |~ "(?i)(error|fail|panic)" # everything, any host, errors
{container="jellyfin"} | json | level="Error" # parse JSON logs, filter a field
sum by (container) (rate({job="docker"} |= "error" [5m])) # error rate per container, as a graph

Pick Last 7 days and watch the histogram above the lines — spikes are incidents.

Add a Logs panel to the server dashboard (90) with {job=~".+"} |~ "(?i)error" — the last errors from anywhere, under the CPU graph. Then an alert rule: sum(count_over_time({job=~".+"} |~ "(?i)out of memory" [10m])) > 0 → Telegram/ntfy. Or on a specific thing: Fail2ban bans, Nextcloud "cron did not run", a backup script's "FAILED".

On the Pi, run Alloy the same way with the loki.write URL pointing at http://192.168.1.50:3100 (publish Loki's port on the server, LAN-only via UFW) and host = "pi". Every machine's logs land in the same Grafana, filtered by host.

Loki compresses well: a home server with thirty containers writes tens of megabytes a day, so 30 days is a couple of gigabytes. Watch du -sh loki-data the first week; lower retention_period if it surprises you.

No logs in Grafana. docker compose logs alloy — usually a permissions problem reading docker.sock or the journal (the machine-id mount matters for the journal). Loki's /ready endpoint from inside the network.

Only some containers appear. Containers using a non-default logging driver (not json-file/journald) can't be read by the Docker source.

Queries slow. Too wide a time range with a regex on all streams; filter by label first ({container="x"}), then text.

Loki won't start: permissions. chown 10001:10001 loki-data.

Two containers added to the monitoring stack, a Loki config with 30-day retention, an Alloy config for containers and the journal, a data source in Grafana. "What happened at 3 am on Tuesday" across every container and every host becomes one query with a histogram — and an alert the next time it happens.