2026-07-21
The upgrade finished; the node was still refusing work
Every node reported the new version, so the upgrade looked done — but the control plane had been left cordoned, and four replicas had nowhere to go.
Background
A kubeadm upgrade from 1.32.13 to 1.33.13 had already been carried out across all three nodes. The binaries were in place and every node reported the new version, which is the check that normally ends an upgrade.
Path
- Symptom
- Four replicas stuck Pending after an upgrade that looked complete
- Assumption
- assume the pods themselves are broken
- Measurement
- read the scheduler's own count: 0/3 nodes available, and the node list shows SchedulingDisabled
- Decision
- uncordon, and make the final uncordon an explicit upgrade step
Response
Uncordon the control plane, and stop treating a version check as an upgrade's completion condition. The remaining scheduling capacity was not incidental: the worker failed the workload's affinity and the third node carried a dedicated taint, so the cordoned control plane was the only viable node and 0 of 3 were available. The separate wave of CrashLoopBackOff and ImagePullBackOff pods across half the namespaces was deliberately left alone — it was the sandbox and registry restart window after a node upgrade, and it cleared itself within minutes.
Skill upgrade
The operating skill now requires the final uncordon to be confirmed explicitly rather than assumed from the upgrade run, and records the post-upgrade pod-flapping window as expected behaviour not to intervene in.
Related system