2026 / 01
Running the full Camunda stack
OutcomeIdentified connection reuse — not connection age — as the cause, and removed an unverified policy instead of layering another tuning fix on top of it.
Overview
Operating a full Camunda 8 orchestration stack (Zeebe, Operate, Tasklist, Optimize, Web Modeler, Keycloak) behind one unified public domain, and diagnosing a connection-handling "fix" that was quietly making things worse.
Context
External access felt slow after a connection policy had been added across 14 single-replica services.
System
Tasklist182 requests / 88 connections
commonHttpProtocolOptions.maxRequestsPerConnection: 1The fix was the problem.
Evidence over guesswork.Problem
A "external access feels sluggish" report led back to a connection-reuse policy that had been added weeks earlier as an unconfirmed judgment call, forcing near-1:1 connection-to-request ratios on 14 backend services.
Investigation
Initial hypothesis
Fresh connections would avoid stale routing and improve external access.
Verified the actual effect directly against Envoy's /clusters stats rather than trusting the original rationale: the "fixed" service showed 697 connections for 778 requests, versus a comparable excluded service reusing connections normally at 88:182. All 14 targeted services were single-replica, so there was never a load-balancing justification for forcing new connections per request in the first place.
Evidence
commonHttpProtocolOptions:
maxRequestsPerConnection: 1Envoy measured 697 connections for 778 requests; Tasklist reused 88 connections for 182 requests.
Rejected fix
Forcing a new connection per request (maxRequestsPerConnection: 1) did remove the failure pattern, but measurement showed it did so by destroying connection reuse entirely — 697 connections for 778 requests, against Tasklist serving 182 requests over 88. The fix worked for the wrong reason, and its cost was unnecessary connection churn across 14 services.
Decision
Delete the unverified policy instead of adding another tuning layer.
Result
Removed outright rather than re-scoped, with the reasoning and the Envoy evidence written down so the same "unconfirmed judgment call" pattern doesn't get silently reintroduced.
What I would change
Current limitation: The comparison is operational evidence from this cluster, not a universal performance benchmark.
Next step: Retain a small repeatable Envoy-stat capture for future routing changes.
Project timeline
2026.10.16
Camunda and OpenSearch move to a London node
Verified on 17 October: Camunda 8.10 and its OpenSearch run on a two-node Oracle Cloud cluster in London, reached from the home cluster over WireGuard. The cutover ran as approved, recorded changes; the Tokyo copies are stopped with their volumes kept. A Zeebe disk-flush saturation found on the way was fixed in configuration, not by a hand patch.
2026.10.09
Camunda 8.10 rebuilt from declared state
Verified on 9 October: the lab's orchestration stack was clean-installed on Camunda 8.10, chart 15.0.0, from a Git-authored rebuild kit. Its own workflows were down for about six minutes; state was re-provisioned from declared files, with no backup by decision.
2026
Camunda 8, full stack, running live
The full orchestration stack behind one unified OIDC domain: the live 8.8 → 8.9 upgrade remains history; production now runs 8.10.