2026 / 02
Running the cluster this whole lab sits on
OutcomeOne delivery path and one internal naming scheme across the registered applications; they discover themselves from their own repositories and retire the same way. Public identity policy is explicit, and existing metrics raise alerts instead of sitting unread.
Overview
Three bare-metal nodes running Kubernetes v1.37.0; 39 ArgoCD applications were Healthy and Synced on 9 September 2026. Applications are discovered from repositories rather than registered by hand, then take the same route: test, build, scan and a separate deployed-tag commit.
Context
The same cluster must support both external requests and repeatable application delivery.
System
Synced is not always deployed.
Problem
Debugging a node from inside a kubectl debug pod is not the same as debugging the node itself: chroot /host is the debug pod's own mount/PID namespace snapshot, not the host's live state — /proc reads empty, and mount/umount run inside it don't propagate to the real host mount table. Trusting that was the difference between a clean fix and a real outage during a NAS rebuild.
Investigation
Initial hypothesis
The debug container represented the live host, and ArgoCD Synced proved the requested build was deployed.
Any operation that depends on genuinely-current host state (writing after a mount, checking whether something is already mounted) has to be handed off to the host's own service manager, not executed directly inside the debug pod's snapshot. The same "verify against the real source of truth, not the convenient proxy for it" habit shows up cluster-wide: a two-stage GitOps pipeline (push triggers a build, a separate later commit bumps the deployed image tag) means checking ArgoCD Synced+Healthy against your own commit proves nothing if that commit never touched an image reference — the check has to wait for the actual tag-bump commit. Two more instances of the same habit turned up later: a password gate that issued its session cookie *before* redirecting to the identity check (so skipping the redirect opened everything — the redirect was a suggestion, not a gate), and alert rules written against a metric name that was never actually being collected. Both were caught by measuring rather than reasoning.
Evidence
# delegated to the host service manager, not run inside the debug containerThe debug pod had a different namespace snapshot; the build commit also preceded a separate tag-bump commit.
Decision
Delegate host-state work to the host's own service manager rather than the debug container, and treat the tag-bump commit as deployment truth.
Result
A LAN-wide wildcard DNS (dnsmasq) plus Cloudflare Zero Trust private routing means every internal *.local hostname is reachable the same way whether you're on the LAN or roaming, with no per-app reconfiguration — documented as the deliberate alternative to exposing raw cluster IPs.
What I would change
Current limitation: Three bare-metal nodes are still a small failure domain, and one of them carries a dedicated taint — so a single node going down can leave a workload with nowhere to schedule. Maintenance still needs deliberate sequencing.
Next step: Improve recovery rehearsal and platform-level service objectives.