2026 / 04
Running the cluster this whole lab sits on
OutcomeOne GitOps delivery path and one internal name resolution scheme across seven services, with node-debugging and deployment-verification pitfalls documented after real incidents.
Overview
Two bare-metal nodes running seven ArgoCD-managed applications behind one shared Gateway API gateway, with a two-stage GitOps pipeline used identically across every app.
Context
The same cluster must support both external requests and repeatable application delivery.
System
Synced は、必ずしもデプロイ完了ではない。Synced is not always deployed.
Problem
Debugging a node from inside a kubectl debug pod is not the same as debugging the node itself: chroot /host is the debug pod's own mount/PID namespace snapshot, not the host's live state — /proc reads empty, and mount/umount run inside it don't propagate to the real host mount table. Trusting that was the difference between a clean fix and a real outage during a NAS rebuild.
Investigation
Initial hypothesis
chroot /host represented the live host, and ArgoCD Synced proved the requested build was deployed.
Any operation that depends on genuinely-current host state (writing after a mount, checking whether something is already mounted) has to be handed off to the host's own systemd via chroot /host systemd-run --wait --pipe --collect, not executed directly inside the debug pod's snapshot. The same "verify against the real source of truth, not the convenient proxy for it" habit shows up cluster-wide: a two-stage GitOps pipeline (push triggers a build, a separate later commit bumps the deployed image tag) means checking ArgoCD Synced+Healthy against your own commit proves nothing if that commit never touched an image reference — the check has to wait for the actual tag-bump commit.
Evidence
chroot /host systemd-run --wait --pipe --collect …The debug pod had a different namespace snapshot; the build commit also preceded a separate tag-bump commit.
Decision
Delegate host-state work through host systemd, and treat the tag-bump commit as deployment truth.
Result
A LAN-wide wildcard DNS (dnsmasq) plus Cloudflare Zero Trust private routing means every internal *.local hostname is reachable the same way whether you're on the LAN or roaming, with no per-app reconfiguration — documented as the deliberate alternative to exposing raw cluster IPs.
What I would change
Current limitation: Two bare-metal nodes remain a small failure domain; maintenance still needs deliberate sequencing.
Next step: Improve recovery rehearsal and platform-level service objectives.
Related notes
chroot /host was not the live node →Stack
Kubernetes · Gateway API (kgateway) · ArgoCD · GitLab CI · Cloudflare Tunnel + Zero Trust · dnsmasq