2026 / 02
Running the cluster this whole lab sits on
OutcomeOne delivery path and one internal naming scheme across the registered applications; they discover themselves from their own repositories and retire the same way. Public identity policy is explicit, and existing metrics raise alerts instead of sitting unread.
Overview
Bare-metal Kubernetes infrastructure and an Oracle Cloud cluster in London host the lab workloads. Dated infrastructure observations are published on the architecture page. Applications are discovered from repositories rather than registered by hand, then take the same route: test, build, scan and a separate deployed-tag commit.
Context
The same cluster must support both external requests and repeatable application delivery.
System
Synced is not always deployed.
Problem
Debugging a node from inside a kubectl debug pod is not the same as debugging the node itself: chroot /host is the debug pod's own mount/PID namespace snapshot, not the host's live state — /proc reads empty, and mount/umount run inside it don't propagate to the real host mount table. Trusting that was the difference between a clean fix and a real outage during a NAS rebuild.
Investigation
Initial hypothesis
The debug container represented the live host, and ArgoCD Synced proved the requested build was deployed.
Any operation that depends on genuinely-current host state (writing after a mount, checking whether something is already mounted) has to be handed off to the host's own service manager, not executed directly inside the debug pod's snapshot. The same "verify against the real source of truth, not the convenient proxy for it" habit shows up cluster-wide: a two-stage GitOps pipeline (push triggers a build, a separate later commit bumps the deployed image tag) means checking ArgoCD Synced+Healthy against your own commit proves nothing if that commit never touched an image reference — the check has to wait for the actual tag-bump commit. Two more instances of the same habit turned up later: a password gate that issued its session cookie *before* redirecting to the identity check (so skipping the redirect opened everything — the redirect was a suggestion, not a gate), and alert rules written against a metric name that was never actually being collected. Both were caught by measuring rather than reasoning.
Evidence
# delegated to the host service manager, not run inside the debug containerThe debug pod had a different namespace snapshot; the build commit also preceded a separate tag-bump commit.
Decision
Delegate host-state work to the host's own service manager rather than the debug container, and treat the tag-bump commit as deployment truth.
Result
A LAN-wide wildcard DNS (dnsmasq) plus Cloudflare Zero Trust private routing means every internal *.local hostname is reachable the same way whether you're on the LAN or roaming, with no per-app reconfiguration — documented as the deliberate alternative to exposing raw cluster IPs.
What I would change
Current limitation: Three bare-metal nodes are still a small failure domain. Camunda runs two Zeebe brokers with two replicas, so rebooting either node that hosts one stops the engine — as the 2026-10 kernel reboots showed. Maintenance still needs deliberate sequencing.
Next step: Improve recovery rehearsal and platform-level service objectives.
Project timeline
2026.10.03
Every node on Ubuntu 26.04 LTS
A fresh compute node piloted the release; the worker and the control plane followed through 24.04 to 26.04.1. All three now run kernel 7.0 with Kubernetes 1.37 and Calico 3.32, and the control plane’s wired NIC moved from an out-of-tree driver to the in-kernel one.
2026
The cluster comes up
Three bare-metal nodes, self-hosted GitLab, ArgoCD — the GitOps foundation everything else ships on.