2026-07 / 06
Harumi Platform — retiring three admin tools into one
OutcomeConsolidated three separate operational surfaces into one controlled console, without discarding the data that lived in the tools being retired.
Overview
A single console for cluster health, resources, docs and connection info — replacing three separate admin tools that had each accumulated a different slice of the same operational picture.
Context
The console needed live ArgoCD data, but the only documented admin credential no longer authenticated, and a scoped read-only account had already been provisioned with no way yet to mint it a token.
System
Role: get,list on pods · namespace-scoped, no secrets退役は、削除ではない。
Retire the surface, not the data.
3 ops surfaces → 1 console, zero downtime.Problem
Three tools — a docs site, an old Camunda dashboard, and the infra repo's own notes — each held a piece of "how is the lab doing right now," and none of them was the one place to check. Retiring the oldest of them (rather than just adding a fourth tool) meant first confirming nothing else depended on what lived inside it — which turned out to be false in a way that mattered.
Investigation
Initial hypothesis
A token could be hand-signed offline with the server's own HMAC secret for the existing read-only account, skipping the admin password entirely.
Built the console read-only first against live cluster state, then ported each retiring tool's actual content across before touching it — URL directory, per-service connection info, and a disk-health board. Auditing what else lived in the oldest tool's namespace surfaced two pieces of shared infrastructure with nothing to do with its content: the Cloudflare tunnel behind every public hostname in the lab, and a disk-health-monitoring DaemonSet. Both were re-homed onto new pods and confirmed healthy — the tunnel's old and new pods briefly shared one token, by design — before the old namespace was deleted.
Evidence
sub: "harumi-platform:apiKey" # not "harumi-platform"
# registry lookup keyed by jti, not iatTwo claim variants were both rejected as "token is revoked" — the server checks a token's jti against a registry, not its issue time, and a local account's subject needs an ":apiKey" suffix the bare account name doesn't have.
Rejected fix
Hand-signing a token offline with the server's own HMAC secret. Two claim variants were tried and both were rejected as "token is revoked" — the server checks a token's jti against a registry rather than its issue time, and a local account's subject needs an ":apiKey" suffix. Continuing to guess the claim shape would have been reverse-engineering an interface that already had a supported endpoint.
Decision
Reset the admin password through the already-documented bcrypt-patch technique, then generate the token through ArgoCD's own account-token endpoint instead of guessing the claim shape further.
Result
The disk-health agent's Go source turned out to have no copy anywhere once its repository was gone — the daily backup that could have recovered it lives on a machine that was offline. Rather than leave a DaemonSet permanently pinned to an orphaned image, rewrote it from scratch: used a throwaway privileged debug pod on each of the three machines to capture real smartctl JSON output first, then wrote the parser against that — including the one detail that would have silently produced garbage data, a USB-attached drive whose SMART attributes only unlock with an explicit ATA-passthrough flag.
What I would change
Current limitation: The issued token has no expiry — acceptable for a hand-rotated service credential, not a pattern to copy for a wider system.
Next step: Apply the same scoped-RBAC, no-secrets pattern to the planned read-only AI diagnostics layer.
Related notes
A hand-signed token was correctly rejected →Stack
Next.js · Kubernetes RBAC · ArgoCD API · Prometheus · Go · Redis