Production-grade Kubernetes for a household. GitOps with Flux · Automated dependency updates with Renovate · Self-hosted by design
This is the live configuration for a multi-node Kubernetes cluster that runs a household — home automation, security cameras, media, document management, AI workloads, and the operational tooling required to keep it all up. Every change lands in Git first; Flux reconciles the cluster from there, and Renovate keeps dependencies current via PRs.
The repo is GitOps-strict: applications are declared as HelmRelease resources, secrets are pulled from 1Password through External Secrets Operator, and clusters are mostly identical except for app selection and sizing. Operational quirks, durability tiers, and security defaults live alongside the manifests in .agents/instructions/ so the conventions are enforceable, not folklore.
flowchart LR
Dev[👤 Operator] -->|git push| Repo[(📦 GitHub<br/>home-ops)]
Renovate[🤖 Renovate] -.->|automated PRs| Repo
Repo -->|reconciles| Flux[⚙️ Flux]
Flux -->|deploys| Cluster[☸️ Kubernetes<br/>11 nodes · 188 apps]
Cluster --> Ceph[(🪨 Ceph<br/>block · default durable)]
Cluster --> LH[(🐂 Longhorn<br/>+ recurring backups)]
Cluster --> Garage[(🧺 Garage<br/>S3-compatible)]
Cluster --> NFS[(🗄️ NFS<br/>beast / brain · bulk media)]
LH -->|weekly + monthly| NFS
Garage -->|rclone CronJobs| AWS[☁️ AWS S3<br/>Glacier Deep Archive<br/>offsite DR]
classDef store fill:#1e293b,stroke:#475569,color:#e2e8f0
class Ceph,LH,Garage,NFS,AWS store
Storage tiers are picked deliberately per workload — see storage-class.instructions.md for the decision tree.
| Layer | Tool | Role |
|---|---|---|
| OS | CentOS Stream 9 / 10 (+ Ubuntu 24.04 on Spark) | Node operating system |
| Runtime | cri-o + crun (containerd on Spark) | CRI + OCI runtime; Spark is the lone containerd node |
| Kubernetes | v1.35.4 | Control-plane and node version |
| GPU | NVIDIA GPU Operator + Container Toolkit | P40 on worker8 (Pascal, 24 GB); GB10 on Spark (Grace-Blackwell, 128 GB unified) |
| GitOps | Flux2 | Declarative cluster reconciliation |
| Automation | Renovate + GitHub Actions | Dependency PRs, link checks, self-hosted runners |
| CNI | Cilium (eBPF) | Networking, BGP peering, LoadBalancer pool |
| Ingress | Envoy Gateway | L7 gateway / HTTPRoute |
| Service mesh | Istio | mTLS + traffic mgmt for mcp-system |
| Admission | Kyverno | Namespace-delete blast-radius + audit-mode policies |
| DNS | external-dns | Cloudflare + bind9 split-horizon |
| TLS | cert-manager | Let's Encrypt + internal CA |
| Tunnel | cloudflared | Public ingress without exposing home WAN |
| AuthN/Z | Authelia + Envoy extAuth | SSO; per-route SecurityPolicy ext-authz gates apps |
| Secrets | External Secrets Operator + 1Password | 116 ExternalSecrets, zero plain-text in Git |
| VPN | wg-easy | Operator OOB WireGuard access |
| Storage | Rook-Ceph, Longhorn, Garage, direct NFS | Tiered by durability requirement |
| Databases | CloudNative-PG, Dragonfly, Qdrant | 24 Postgres clusters, KV, vector |
| Observability | kube-prometheus-stack, Loki, Tempo, Grafana | Metrics, logs, traces, dashboards |
| Telemetry | OpenTelemetry Collector + Vector | Trace pipeline (→ Tempo) + log shipping (→ Loki) |
| Images | ZOT | Pull-through registry / local cache |
| Role | Hostname | Device | CPU | RAM | OS | Storage / Accelerators | Notes |
|---|---|---|---|---|---|---|---|
| 🧠 | master1 | bare-metal | 4 | 32 GB | CentOS 10 | NVMe (Longhorn) | Intel iGPU · RTL-SDR · control plane |
| 🧠 | master2 | VM on beast | 3 | 10.5 GB | CentOS 9 | virtualized control plane | |
| 🧠 | master3 | VM on beast | 3 | 10.5 GB | CentOS 9 | virtualized control plane | |
| 💪 | worker2 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | ZWA-2 Z-Wave dongle |
| 💪 | worker3 | ThinkCentre M910x | 8 | 64 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | Sonoff Zigbee dongle |
| 💪 | worker4 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | Coral USB TPU |
| 💪 | worker5 | VM on beast | 10 | 21.5 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | |
| 💪 | worker6 | VM on beast | 10 | 26.5 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | |
| 💪 | worker7 | VM on beast | 10 | 26.5 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | |
| 🎮 | worker8 | VM on beast | 10 | 51.5 GB | CentOS 9 | NVMe (Longhorn + Ceph OSD) | NVIDIA P40 (24 GB VRAM) |
| 🚀 | spark | NVIDIA DGX Spark | 20 | 128 GB | Ubuntu 24.04 | NVMe + 8 GPU slots | NVIDIA GB10 (Grace-Blackwell, 128 GB unified); arm64 · containerd outlier |
| Host | Role |
|---|---|
beast |
Dell R730xd · iDRAC 8 · RAID6 bulk storage · primary NFS · Longhorn backup target · Garage substrate · VM host |
brain |
Router/gateway · RAID6 mass_storage · NFS for downloads & TV · OOB SSH on :3231 |
Physical topology (click to expand)
| Network | CIDR | VLAN |
|---|---|---|
| Default | 192.168.0.0/16 |
0 |
| IoT | 10.10.20.0/24 |
20 |
| Guest | 10.10.30.0/24 |
30 |
| Security (cameras) | 10.10.40.0/24 |
40 |
| Kubernetes pod subnet (Cilium) | 10.42.0.0/16 |
— |
| Kubernetes services subnet (Cilium) | 10.43.0.0/16 |
— |
| Kubernetes LB pool (CiliumLoadBalancerIPPool) | 10.45.0.0/24 |
— |
Worker nodes attach to iot and sec VLANs via Multus for direct camera and IoT-device reachability. Cilium peers BGP with the upstream router to advertise the LB pool; external ingress flows through Envoy Gateway behind cloudflared.
🏠 Home Automation — Home Assistant ecosystem, 400+ devices
| App | Purpose |
|---|---|
| Home Assistant | Primary orchestrator; 400+ Z-Wave / Zigbee / Matter / ESPHome devices |
| ESPHome | Build & deploy firmware for DIY sensors |
| EMQX | MQTT broker |
| Node-RED | Visual automation flows |
| Zigbee2MQTT | Zigbee bridge (Sonoff stick on worker3) |
| Z-Wave JS UI | Z-Wave bridge (ZWA-2 stick on worker1) |
| Matter Server | Matter protocol bridge |
| Frigate | NVR + ML camera analysis (7+ cameras, Frigate+ trained model) |
| NetBox | IPAM / DCIM |
| wyoming-services | Piper TTS + Whisper STT for voice |
| smtp-relay | Maddy → Mailgun outbound mail |
🎬 Media & Entertainment — Jellyfin, Immich, Music Assistant, RomM
| App | Purpose |
|---|---|
| Jellyfin | Primary media server (read-only metadata) |
| Immich + immich-pet-tagger + immichkiosk + immich-power-tools | Photo library with ML face/pet recognition, offsite-backed |
| Music Assistant + Gonic | Multi-room music control + Subsonic API |
| RomM | Retro game library (~10k ROMs) |
| Beets | Music library tagging |
| cutVideo / av1corrector / videodupfinder / medialyze | Custom video tooling |
| Theme Park | Consistent UI theming across apps |
| Batocera Webdashboard Pro | Retro-gaming console dashboard |
| kodi-playback-watcher | Bridge for Kodi playback state |
🤖 AI & ML — Local inference, agents, image generation (namespace ai/)
| App | Purpose |
|---|---|
| Ollama (P40) | Local LLM serving on the Pascal P40 (≤8b-class models, embeddings) |
| Ollama Spark | LLM serving on Spark/GB10 (qwen3-next:80b-a3b-instruct-q4_K_M for Open WebUI, bge-m3 embeddings) |
| ComfyUI | Image generation workflows |
| Khoj | Personal AI assistant over notes + docs (Authelia extAuth-gated) |
| Paperless-AI | Auto-tagging for paperless-ngx |
| tei-spark | Text-embedding-inference reranker (unsuspended 2026-05-21) |
📊 Observability — Prom/Loki/Grafana
| App | Purpose |
|---|---|
| kube-prometheus-stack | Prometheus + AlertManager + node-exporter |
| Loki | Log aggregation |
| Tempo | Distributed tracing backend (SingleBinary mode) |
| OpenTelemetry Collector | Trace ingestion pipeline (apps → OTel → Tempo) |
| Vector | Log shipping (sources → Loki) |
| Grafana | Dashboards + alerting UI |
| kube-state-metrics / kube-ops-view | Cluster state & visualization |
| Goldilocks | VPA-driven resource right-sizing recommendations |
| Kromgo | Prometheus → Glance dashboard bridge |
| Netdata | Per-node real-time metrics |
| network-ups-tools (NUT) | UPS monitoring & graceful shutdown |
| exporters | Custom Prometheus exporters |
🗄️ Data & Storage — Databases, object storage, vector search
| App | Purpose |
|---|---|
| CloudNative-PG | 24 Postgres clusters with WAL archiving to Garage |
| Dragonfly | Redis-compatible in-memory store |
| Qdrant | Vector DB for embeddings / RAG |
| pgAdmin | Postgres admin UI |
| Rook-Ceph | Distributed block storage (default durable tier) |
| Longhorn | Block storage with NFS-backed recurring backups |
| Garage | S3-compatible object storage (DB backups, app S3 workloads) |
🌐 Network, Auth & Platform — Ingress, SSO, GitOps machinery
| App | Purpose |
|---|---|
| Cilium | CNI, BGP, LoadBalancer pool |
| Envoy Gateway | Ingress / HTTPRoute (30 routes) |
| cert-manager | TLS certificate lifecycle |
| external-dns | Cloudflare + bind9 record sync |
| cloudflared | Public tunnel without exposed WAN |
| Authelia | OIDC identity provider |
| LLDAP | Lightweight LDAP directory backing Authelia |
| extAuth SecurityPolicies | per-route Authelia ext-authz gating app SSO |
| wg-easy | Primary OOB WireGuard access |
| External Secrets Operator | 1Password-backed secret materialization |
| Flux2 | GitOps reconciler |
| Renovate | Image & Helm chart update PRs |
| Kuadrant | MCP server gateway (Authelia-gated JWT) |
| Kyverno | Admission controller — namespace-delete blast-radius + audit-mode policies |
| actions-runner-controller | Self-hosted GitHub Actions runners |
| ZOT | Pull-through registry cache |
🗂️ Documents & Collaboration — Personal knowledge stack + self-hosted tools
| App | Purpose |
|---|---|
| Paperless-ngx | Document scanning, OCR, tagging (CNPG-backed, offsite-backed) |
| Obsidian + obsidian-couchdb | Notes sync (CouchDB w/ Cloudflare rate-limiting) |
| Zulip | Self-hosted team chat |
| Windmill | Workflow automation; 7 checked-in TypeScript flows under kubernetes/apps/home/windmill/workflows/ cover paperless RAG ingest+tombstone (Qdrant), LightRAG graph-RAG ingest+tombstone, HA smart-home intent drift, Windmill self-failure-watch, and the workaround upstream-watcher. (Down from 23 — the 16 langgraph-agents-fleet-specific flows, including the inbox/approval/digest/DLQ/cost-cap/awaiting-user/reviewer-weekly scripts and the Zulip triager webhook, were removed 2026-07-06 along with the fleet.) |
| ntfy | Self-hosted push notifications (operator approvals via Android tap actions) |
| BentoPDF | Self-hosted PDF toolkit |
| Kitchenowl | Shopping lists + recipe / meal management |
| Open WebUI | Self-hosted LLM frontend; routes chat to Ollama-Spark (default) / Ollama-P40, and pulls in the MCP gateway as its only tool server (the langgraph-agent-as-model registration and the HolmesGPT tool server were both removed 2026-07-06). RAG via bge-m3 + bge-reranker-v2-m3 over Qdrant |
| SearXNG | Privacy-respecting metasearch engine |
| Glance | Personal dashboard / start page |
| Atuin | Encrypted shell-history sync across machines |
| IT-Tools | Self-hosted developer toolbox |
| MediKeep | Personal medical records |
| Nametag | Name tag / badge generator |
| Pump + Pump-cv | Custom personal apps (rwlove-built) |
🔌 MCP Servers — 20 Model Context Protocol servers behind an Authelia-gated gateway
| Server | Exposes |
|---|---|
| mcp-gateway | Aggregating gateway; Envoy SecurityPolicy validates Authelia-issued JWTs (daily-rotated key) |
| ha-mcp | Home Assistant entities + service calls |
| esphome-mcp | ESPHome dashboard: device YAML edit, validate, logs, compile + OTA flash |
| immich-mcp | Immich library search + asset metadata |
| kubectl-mcp | Cluster introspection + safe kubectl ops |
| grafana-mcp | Grafana dashboards + Loki/Prom queries |
| prometheus-mcp | Direct PromQL access |
| paperless-mcp | Paperless-ngx document search |
| netbox-mcp | NetBox IPAM / DCIM |
| github-mcp | GitHub repo + PR ops |
| omada-mcp | TP-Link Omada controller |
| searxng-mcp | Privacy search through SearXNG |
| arr-mcp | Library-search interface to media-pull apps |
| time-mcp | Time / timezone utilities (rwlove/time-mcp native-SHTTP build) |
| chrome-mcp | Playwright-driven Chromium browser automation for agents |
| memory-mcp | Cross-agent knowledge graph backed by Postgres + pgvector (bge-m3 1024-dim) |
| cilium-mcp | Read-only Cilium / Hubble introspection (kubectl-mcp-style, Cilium-scoped) |
| windmill-mcp | Aggregated Windmill workspace tools (script run, flow trigger) |
| codegraph-mcp | Code knowledge-graph queries (CodeGraphContext via mcp-proxy --stateless) |
Local-first by default — chat and retrieval today; the automated agent fleet and its cloud-escalation gate were both removed 2026-07-06 (see Current state below). Claude Code (this CLI) is available for interactive, operator-driven use, but there is no automated pipeline that reaches it or the Claude API on its own.
📖 Full chapter: see
docs/src/ai_architecture.mdfor per-app integration paths, RAG pipelines, the historical agent-fleet roster, and the file:line references behind every claim here.
flowchart TB
subgraph Surfaces[Surfaces]
OWUI[Open WebUI<br/>chat + RAG]
Khoj[Khoj<br/>personal AI]
end
subgraph Bridges[Windmill bridges<br/>7 TS workflows]
WPaperless[paperless-rag-ingest/tombstone.ts]
WLightrag[lightrag-rag-ingest/tombstone.ts]
WOther[smart-home-intent-drift.ts ·<br/>windmill-failure-watcher.ts ·<br/>workaround-watcher.ts]
end
subgraph Inference[Inference]
OllamaP40[(ollama / P40<br/>qwen2.5:7b · embeddings)]
OllamaSpark[(ollama-spark / GB10<br/>qwen3-next:80b-a3b-instruct-q4_K_M · bge-m3)]
end
subgraph Tools[Tools + retrieval]
Gw[MCP Gateway<br/>20 servers]
Q[(Qdrant<br/>vector DB)]
Mem[(memory-mcp<br/>pgvector KG)]
end
subgraph Outputs[Outputs + observability]
AM[AlertManager]
Push[Pushover<br/>direct page]
end
OWUI -->|chat| OllamaSpark
OWUI --> Gw --> Mem
OWUI --> Q
Khoj --> OllamaP40
WPaperless --> Q
WLightrag --> Q
AM --> Push
There's no agent fleet, no gated Claude API lane, and no OTLP trace
producer left in this diagram — all three were langgraph-agents
plumbing, removed 2026-07-06 along with the fleet.
- Open WebUI (
collab/) — primary chat UI. Defaults to qwen3-next:80b-a3b-instruct-q4_K_M on Ollama-Spark. RAG runs over Qdrant with bge-m3 embeds + BGE reranker-v2-m3 in-process. Tool server wired in: the MCP gateway (the langgraph-agent-as-model registration and the HolmesGPT tool server are both gone). - Khoj (
ai/) — parallel personal-AI surface for notes + docs. Self-contained: own embedding pipeline (default gte-small, optionally ollama nomic-embed-text), chat via Ollama-P40. Does not consume the MCP gateway. - HolmesGPT — removed 2026-07-06 (no value delivered). The
windmill-investigateroute/receiver and thealertmanager-holmesgpt-notify.tsbridge were removed in Stage 1 (critical alerts now go straight to Pushover); the HolmesGPT deployment itself, its RBAC, its Open WebUI tool-server registration, and its Ollama/Authelia wiring were removed in Stage 2. HolmesGPT no longer exists anywhere in the cluster. - langgraph-agents — removed 2026-07-06. The FastAPI multi-agent runtime, its Postgres checkpoints (
postgres-langgraph-checkpoints, deleted), vault PVCs (langgraph-vault/langgraph-vault-rw, deleted), and both public routes (hai.${SECRET_DOMAIN}andhai-web.${SECRET_DOMAIN}) are gone.sync-receiver— an sshd sidecar that existed solely to expose the langgraph-vault PVCs over rsync — was deleted alongside it. See Agent fleet — status today for the historical roster. - Windmill (
home/) — 7 checked-in TypeScript flows remain underkubernetes/apps/home/windmill/workflows/(down from 23): Paperless RAG ingest+tombstone, LightRAG graph-RAG ingest+tombstone, HA smart-home intent drift, Windmill self-failure-watch, and the workaround upstream-watcher. The 16 langgraph-fleet-specific flows (inbox, approval post/receive, daily digest, cost-cap watcher, awaiting-user sweep, DLQ watcher, weekly operator-drift crons, the approval-flow smoke test, and the Zulip triager webhook) were deleted 2026-07-06. - Langfuse — removed 2026-07-06. Its only consumer (langgraph-agents) was already gone; the keep-dormant-vs-remove question is now resolved as remove.
- memory-mcp (
mcp-system/) — cross-agent knowledge graph onpostgres-langgraph-memorywith pgvector(1024), bge-m3 embeds via Ollama-Spark. Fully unaffected by the decommission — it's memory-mcp's own database, used by Claude Code and Open WebUI. It just lost langgraph-agents as a consumer.
The langgraph-agents multi-agent fleet (triager, supervisor,
historian, reporter, reviewer, the five operator personas —
storage/network/observability/ml/smart-home — plus
errand-runner, researcher, homelab-engineer, and eight more
wired-but-cold agents) was removed in its entirety. There is no
automated agent pipeline in the cluster today. The full historical
roster with per-agent role and trigger detail lives in
docs/src/ai_architecture.md,
preserved as a record of what was built and retired — not as a status
board for anything currently running.
| Tier | Backend | When used |
|---|---|---|
| 1 | qwen2.5:7b on Ollama (P40) |
Fast / simple chat, embeddings, voice STT/TTS support |
| 2 | qwen3-next:80b-a3b-instruct-q4_K_M on Ollama-Spark (GB10) |
Default chat inference (Open WebUI) |
Removed 2026-07-06: the third tier — gated Claude API escalation
via langgraph-agents (ENABLE_CLAUDE_API, cost caps $5/task ·
$10/agent/day · $30/global/day) — is gone along with the fleet.
There is no automated cloud-escalation tier today.
Before 2026-07-06, holding the phone's power button and saying "inbox
<whatever I'm thinking>" routed through Whisper STT → an HA
rest_command → Windmill → langgraph-agents /inbox, which
triaged the note to a specialist agent and synced results to the
Obsidian vault. That backend is gone.
Known gap, not yet fixed: the HA voice rest_command (defined in
the separate home-assistant-config repo) still exists and still
POSTs toward the deleted Windmill webhook — it will silently fail.
The full original design (mermaid diagram + step-by-step path) is
preserved as a historical record in
docs/src/workflow_automation.md.
There is no automated investigation step today. AlertManager fires a critical alert and pages Pushover directly. HolmesGPT used to sit in between (querying Prometheus, Loki, and the cluster to build a root-cause hypothesis before posting back via Windmill) but was removed 2026-07-06 — no value delivered.
- HolmesGPT — removed. No longer deployed anywhere in the cluster.
- LangGraph fleet — removed in its entirety: the FastAPI runtime,
postgres-langgraph-checkpoints, thelanggraph-vault/langgraph-vault-rwPVCs, both public routes (hai.${SECRET_DOMAIN},hai-web.${SECRET_DOMAIN}), andsync-receiver. 16 of 23 Windmill workflows deleted with it.postgres-langgraph-memoryandmemory-mcpare kept — that database is memory-mcp's own backend, unaffected by this decommission. Known gap: the HA voice "inbox …" intent is now broken (see Voice-to-action above). - claude-runner — retired 2026-05-23, superseded by the langgraph fleet (its two CronJobs graduated into agent workflows inside langgraph-agents). With langgraph-agents now also removed, that functionality is gone twice over — there is no automated Claude Code or Claude API pipeline anywhere in the cluster today. Claude Code use is interactive-only (an operator running the CLI, same as the session that made this edit).
- KubeClaw — retired (memo
project_open_issues_cleanup_2026_05_20).
| Service | Use | Cost |
|---|---|---|
| 1Password | Secret backend for External Secrets | ~$65 / yr |
| Cloudflare | Domain, DNS, tunnel, WAF rate-limiting | Free |
| GitHub | Repo hosting + CI | Free |
| Mailgun | Outbound mail relay (via Maddy) | Free (Flex) |
| Pushover | Push notifications for AlertManager + apps | $10 one-time |
| Frigate+ | Trained ML model for Frigate NVR | $50 / yr |
| AWS S3 Glacier Deep Archive | Offsite DR for Immich + Paperless (objects + DB backups) | ~$1–5 / mo (varies) |
| ~$10–15 / mo |
Four tiers, picked by what the data has to survive — node loss, Ceph loss, cluster loss, or full site loss. Databases get ceph-block + Barman→Garage; irreplaceable state goes to Longhorn with NFS-shipped weekly + monthly backups; S3-shaped workloads use Garage; bulk media rides direct NFS. Full decision tree: .agents/instructions/storage-class.instructions.md.
All 116 ExternalSecrets resolve through External Secrets Operator from 1Password. Application credentials are templated into ExternalSecret resources and never live in YAML. Cross-namespace mirrors use the reflector pattern when consumer charts hard-code secret names.
Authelia (with LLDAP) is the identity provider; per-route Envoy Gateway SecurityPolicy extAuth (Authelia's ext-authz endpoint) enforces auth at the gateway — 26 apps sit behind SSO today, tiered admin vs household via access_control rules (the oauth2-proxy sidecar fleet was retired 2026-07-01, #12767). Native-OIDC apps (immich, paperless, open-webui, …) authenticate against Authelia directly. The mcp-gateway validates Authelia-issued JWTs with a daily-rotated signing key for MCP tooling.
kube-prometheus-stack scrapes everything; Loki ingests pod logs (via Vector); Tempo ingests traces (via OpenTelemetry Collector); Grafana stitches the dashboards. AlertManager fans critical alerts straight to Pushover — no automated triage step (the HolmesGPT investigation path was removed 2026-07-06).
Two GPUs split the workload:
- NVIDIA P40 on worker8 (Pascal, 24 GB VRAM) — Ollama for ≤8b-class models + embeddings, ComfyUI, Whisper STT, Immich CLIP face/pet recognition, and the immich-pet-tagger fork pinned to a P40-compatible PyTorch build.
- NVIDIA GB10 on Spark (Grace-Blackwell, 128 GB unified) — the larger Ollama deployment serving qwen3-next:80b-a3b-instruct-q4_K_M for Open WebUI, plus bge-m3 embeddings for the cross-agent knowledge graph and Paperless RAG.
Driver lifecycle is handled by the NVIDIA GPU Operator. Spark is the lone containerd node in an otherwise CRI-O cluster; a NodeFeatureRule auto-skips the GPU container-toolkit DaemonSet on CRI-O nodes.
Per-app rclone CronJobs ship Immich originals and Paperless documents — plus their Garage-stored Postgres backups — to encrypted AWS S3 with a 1-day Glacier Deep Archive transition. Recovery procedure is documented at Offsite recovery and was last validated 2026-05-05.
Every change reaches the cluster through Git. Flux suspends are a deliberate manual signal — paused Kustomizations are not "broken," they're intentional pauses for in-flight maintenance and are documented in conventions, not reverted on sight.
The full operator handbook lives at https://rwlove.github.io/home-ops/.
Frequently referenced pages:
- AI architecture
- Cluster rebuild
- Initialization & teardown
- Cluster upgrade
- Power outage recovery
- Limits & requests philosophy
- Debugging playbook
- Offsite recovery
- Immich restore to new CNPG database
- NVIDIA P40 GPU setup
- master1 etcd disk swap
- GitHub webhook
Repo-local conventions (auto-loaded by AI agents from .agents/instructions/):
- Storage class selection · HelmRelease security defaults · ConfigMap layout · Sorting rules · Schema correction · Persona
Inspired by the k8s-at-home community. @whazor maintains the excellent k8s-at-home search — a great way to discover how others configure the same Helm releases.