HARUMI
Back

2026 / 03

Running the full Camunda 8.8 stack

Home lab infrastructureSole operator — investigation and implementation

OutcomeIdentified connection reuse — not connection age — as the cause, and removed an unverified policy instead of layering another tuning fix on top of it.

Overview

Operating a full Camunda 8 orchestration stack (Zeebe, Operate, Tasklist, Optimize, Web Modeler, Keycloak) behind one unified public domain, and diagnosing a connection-handling "fix" that was quietly making things worse.

Context

External access felt slow after a connection policy had been added across 14 single-replica services.

System

778REQUESTS
697CONNECTIONS

Tasklist182 requests / 88 connections

commonHttpProtocolOptions.maxRequestsPerConnection: 1

問題は、追加した「修正」だった。

The fix was the problem.

Evidence over guesswork.

Problem

A "external access feels sluggish" report led back to a connection-reuse policy that had been added weeks earlier as an unconfirmed judgment call, forcing near-1:1 connection-to-request ratios on 14 backend services.

Investigation

Initial hypothesis

Fresh connections would avoid stale routing and improve external access.

Verified the actual effect directly against Envoy's /clusters stats rather than trusting the original rationale: the "fixed" service showed 697 connections for 778 requests, versus a comparable excluded service reusing connections normally at 88:182. All 14 targeted services were single-replica, so there was never a load-balancing justification for forcing new connections per request in the first place.

Evidence

commonHttpProtocolOptions:
  maxRequestsPerConnection: 1

Envoy measured 697 connections for 778 requests; Tasklist reused 88 connections for 182 requests.

Rejected fix

Forcing a new connection per request (maxRequestsPerConnection: 1) did remove the failure pattern, but measurement showed it did so by destroying connection reuse entirely — 697 connections for 778 requests, against Tasklist serving 182 requests over 88. The fix worked for the wrong reason, and its cost was unnecessary connection churn across 14 services.

Decision

Delete the unverified policy instead of adding another tuning layer.

Result

Removed outright rather than re-scoped, with the reasoning and the Envoy evidence written down so the same "unconfirmed judgment call" pattern doesn't get silently reintroduced.

What I would change

Current limitation: The comparison is operational evidence from this cluster, not a universal performance benchmark.

Next step: Retain a small repeatable Envoy-stat capture for future routing changes.

Related notes

697 connections for 778 requests

Stack

Camunda 8.8 · Zeebe · Keycloak · Elasticsearch · Envoy / kgateway

Next project

Running the cluster this whole lab sits on