Local High Availability

Local high availability runs multiple Eden nodes against one shared, authoritative Postgres store. Choose active-passive for the simplest recovery model or active-active when every healthy node should serve live traffic.

Active-Passive

The primary receives all traffic. Safe read-replica routing is optional, and a standby receives no live traffic. Recovery requires promoting a passive node and changing DNS, the Kubernetes Service target, or EDEN_LB_PRIMARY.

Active-Active

Pingora distributes reads and writes across all healthy active nodes. It probes GET /health and removes a failed node after the configured consecutive failures. Active-active cache peers use stable identities, direct per-shard addresses, and mutually authenticated TLS.

Compose uses service DNS names. Helm uses a StatefulSet and headless Service; Eden re-resolves those peer names every EDEN_HA_CACHE_SYNC_PEER_REFRESH_INTERVAL_MS (default 5000ms) so a replacement pod can receive a new IP without leaving the surviving node pinned to the old route.

Cache Ordering

Postgres remains the source of truth. Committed RBAC cache values, explicit revocation markers, and endpoint or organization fences replicate directly through ShardKV. Each record carries the Postgres (version_ms: i64, version_seq: i64) tuple as a fixed 16-byte ordered revision. A higher Postgres revision wins even when packets arrive out of order.

The revision does not use a wrapping u8. Although serial arithmetic can treat 255 as earlier than 0, ordering becomes ambiguous at 128 revisions of lag and values repeat after 256 writes. An HA peer can be disconnected longer than that. Eden therefore treats revision 255 as earlier than 256 without wrapping.

If either revision component is i64::MAX, Eden keeps the revision as an ordering boundary but does not serve that cached authorization value. The read falls back to authoritative Postgres. A later greater tuple restores caching only when neither component is still i64::MAX (for example, a later timestamp can follow a maximal sequence). Any tuple whose timestamp is i64::MAX, including the fully maximal tuple, remains a read boundary until the cache generation is replaced. It never wraps to i64::MIN.

Eden publishes a replicated RBAC value only when Postgres reports that the revision was applied. Stale or equal mutations rejected by Postgres cannot overwrite the cache.

RBAC writes must go through Eden's write methods. Direct SQL writers are not a supported HA path because RBAC triggers do not enqueue unversioned invalidation events; an additional writer needs a revisioned database changefeed before it can safely update the replicated cache.

Entity ID/UUID mappings, ELS, encryption metadata, and VPN identity records do not all expose a Postgres revision yet. A compatibility outbox invalidates those process-local cache families on every HA node in active-passive and active-active mode. It does not clear the replicated RBAC map.

Health

/health deliberately remains healthy when a peer dies, allowing the survivor to stay in the traffic pool. /health/cache-sync reports 503 while peers are joining or failing. During that state, authorization and membership cache reads bypass ShardKV and use Postgres. Persistent gateway connections revalidate on every request while the cache version is indeterminate.

Recovery And Annual Planning

With the default 1000ms Pingora interval and 3 failed checks, p99 failure detection is 2990ms and the maximum is 3000ms per incident. This is not three seconds of downtime per month or year; it is the routing recovery cost each time a node fails.

Here p99.9 and similar labels denote annual server uptime targets, not request latency percentiles. Assuming ten independent node incidents and a healthy Pingora and Postgres tier:

These figures exclude shared Postgres or Pingora outages, correlated failures, deployments, configuration errors, and clients without retries.

Deploy

Active-passive Compose:

EDEN_JWT_SECRET=... \
CARGO_FEATURES=server-runtime,high-availability \
docker compose -f docker-compose.yaml -f docker-compose.ha.yaml \
  --profile ha --profile balancer up --build

For active-active, add docker-compose.ha.active-active.yaml and set EDEN_HA_CACHE_SYNC_CERT_DIR to a directory containing public/ plus one private identity directory per node. Each container receives the public certificates and only its own private key. The Helm equivalent is helm/charts/eden/examples/ha-active-active-values.yaml; it expects separate public and identity Secrets and mounts only the current StatefulSet pod's key.

Validate

EDEN_JWT_SECRET=... scripts/high-availability/check-local-config.sh
scripts/high-availability/test-active-passive-failover.sh
scripts/high-availability/test-active-active-failover.sh