Deploy Eden with high availability

Use this guide for a retained-identity active-passive Kubernetes deployment with external PostgreSQL and ClickHouse. It applies to AKS, EKS and other Kubernetes clusters that meet the release prerequisites. It describes configuration, not a claim that every cloud, storage class or failure scenario has been qualified.

Installation gate: obtain an explicitly qualified Professional or Enterprise HA image and its matching chart before installing. Ordinary node-bound single-instance credentials alone do not authorize distinct physical replicas. HA also requires a signed deployment policy for the paired Service/endpoint credentials and a current physical membership lease. These commands are a configuration reference; a successful Helm render does not prove those conditions or qualify a release. If the qualified HA artifact is not available in Images, contact the Eden team.

An HA deployment needs an HA image and its matching chart. Adding replicas to the embedded single-container image does not create an HA deployment. Never share a Turso database, DuckDB database or outbox volume between active Eden processes.

1. Obtain the deployment inputs

Ask the Eden team for the licensed HA release bundle:

• the matching versioned Helm chart and its packaged HA examples;
• the HA image repository, immutable digest and qualification record;
• the online HA deployment context, paired Service/endpoint credentials and

qualified licensing integration for the deployment and its replicas;

• a stable deployment UUID and the organization/network/HA-group identities;
• cache-sync certificates, production HA identities and a TOML configuration

for each StatefulSet ordinal; and

• the release's PostgreSQL and ClickHouse bootstrap/upgrade procedures.

The single-instance image is a separate artifact. Its successful download or migration test does not qualify an HA image. Standard uses the single-instance profile; HA belongs to the Professional and Enterprise capability sets. Use the exact tier and deployment profile shown in the qualified release metadata.

The current Helm chart rejects offline_lease licensing with HA. It supports offline leases only for single-node installations. Do not reuse a node-bound trial/offline lease across replicas or turn license enforcement off to make a cluster start. Keep the stable licensed identity separate from each pod's physical runtime identity. Sharing one physical node UUID or inventing an unbound key is not a supported workaround.

You also need Helm 3, kubectl, Kubernetes permissions to manage the namespace, an image-pull Secret if required, and customer-managed PostgreSQL and ClickHouse. The Eden chart does not provision production databases, ingress or monitoring.

2. Choose the topology and storage

For the initial customer release use five validators, with four required for control-plane changes. Other sizes below describe the consensus model, not additional release-qualified deployment profiles:

Customer traffic stays on StatefulSet ordinal zero. Recovery stops and fences the old primary, then recreates that same ordinal and identity on a compatible replacement host with its retained disk, keys and license custody. It does not promote another logical migration owner. Distributed lane execution and cross-identity takeover are outside this release; see HA Redis migrations.

Place pods across independent hosts and, where available, availability zones. Keep the configured PodDisruptionBudget consistent with the required quorum; it does not protect against involuntary node loss. PostgreSQL, ClickHouse, ingress and load balancers need their own availability and backup policies.

Each Eden pod needs a separate persistent ReadWriteOnce volume for its coordination stores and audit/telemetry outboxes. Choose a storage class with the required zone placement, retention and attachment behavior. On AKS this usually means a suitable Azure Disk CSI storage class; verify the actual cluster's settings. Do not set one shared existingClaim for multiple pods. Size storage for retained events, outboxes, coordination history and recovery headroom. The example's CPU/memory values are starting requests, not sizing or throughput guarantees.

3. Prepare values and database access

Download the chart version matching your qualified image:

EDEN_CHART_VERSION='<exact-version-from-Images>'
helm repo add eden https://eden.dev/eden/charts
helm repo update eden
mkdir -p eden-install
helm pull eden/eden --version "$EDEN_CHART_VERSION" --untar --untardir eden-install/vendor
EDEN_CHART_PATH=eden-install/vendor/eden

The chart is public; product image downloads still require an eligible license. An unpublished chart version is unavailable. This example names the StatefulSet eden-ha, so its pod names are eden-ha-0, eden-ha-1, and so on. Provision identities for these exact names before installation.

umask 077
mkdir -p eden-ha-install
cp "$EDEN_CHART_PATH/examples/production-ha-values.yaml" eden-ha-install/values.yaml
cp "$EDEN_CHART_PATH/examples/production-active-passive-values.yaml" eden-ha-install/active-passive.yaml
kubectl create namespace eden

Edit eden-ha-install/values.yaml; retain the other production settings from the example. The following shows the current chart fields to review, using five pods. Apply the active-passive overlay in every command below. Replace its five example retainedNodeUuids once before installation, then retain their ordinal order across image updates and host replacement. The endpoint key and paired Service credential remain subject to the installation gate above:

fullnameOverride: eden-ha
deploymentMode: ha
replicaCount: 5
datastore:
  mode: external
image:
  repository: registry.customer.example/eden/service-ha
  digest: sha256:<issued-64-hex-digit-digest>
  requireDigest: true
secretName: eden-application
env:
  POSTGRES_HOST: postgres.customer.example
  POSTGRES_PORT: "5432"
  CLICKHOUSE_URL: https://clickhouse.customer.example:8443
  EDEN_CLICKHOUSE_SCHEMA_MANAGEMENT_ENABLED: "false"
clickhouseCredentials:
  secretName: eden-clickhouse
  usernameKey: username
  passwordKey: password
licensing:
  enforcement: required
  phoneHomeUrl: https://license.eden.dev
  clusterUid: <stable-deployment-uuid>
  validationTtlSecs: 300
  haContext:
    secretName: eden-ha-licensing-context
  licenseKeySecret:
    secretName: eden-license
    secretKey: EDEN_LICENSE_KEY
highAvailability:
  enabled: true
  trafficMode: active_passive
  nodeRole: standby
  nodeUuidFromPodUid: false
  cacheSync:
    enabled: false
    clusterId: customer-eden-production
    shardCount: 8
    portBase: 7100
    publicSecretName: eden-ha-cache-sync-public
    identitySecretName: eden-ha-cache-sync-identities
productionHa:
  enabled: true
  configSecretName: eden-production-ha-config
  identitySecretName: eden-production-ha-identities
  networks:
    - <canonical-network-uuid>
podDisruptionBudget:
  enabled: true
  minAvailable: 4
telemetryOutbox:
  persistence:
    enabled: true
    storageClassName: <customer-storage-class>
    size: 20Gi

Keep five validators and minAvailable: 4 for this profile. Keep the example's topology spread labels aligned with the chart's nameOverride. Supply any registry authentication through imagePullSecrets.

Create private files through your secret manager; keep their contents out of Helm values, shell history and source control:

Retain the encryption key and JWT secret in your recovery store. The chart maps POSTGRES_DB to both PostgreSQL database environment aliases. DBOS uses this PostgreSQL service by default. If DBOS needs a different database, supply EDEN_DBOS_DATABASE_URL using an existing Secret through extraEnvFrom.

Bootstrap the selected databases using the matched release procedures and separate administrative credentials. The ClickHouse runtime user must have the private service role emitted by eden-clickhouse-bootstrap; leave runtime schema management disabled. Configure database TLS/trust using the release's supported connection settings and mounted Secrets. The hostname alone does not configure a TLS policy. Do not use the production application credential as a general database administrator.

chmod 600 eden-ha-install/*.env
kubectl -n eden create secret generic eden-application --from-env-file=eden-ha-install/application.env
kubectl -n eden create secret generic eden-clickhouse --from-env-file=eden-ha-install/clickhouse.env
kubectl -n eden create secret generic eden-license --from-env-file=eden-ha-install/license.env

Use the purchased endpoint credential for EDEN_LICENSE_KEY and its paired Service credential for service.key. Obtain ha-context.json and that Service credential through your approved HA enrollment bundle. The context selects the licensed deployment and organization; it does not itself grant permissions. The running image verifies the signed policy and deployment binding.

chmod 600 eden-ha-install/ha-context.json eden-ha-install/service.key
kubectl -n eden create secret generic eden-ha-licensing-context \
  --from-file=context.json=eden-ha-install/ha-context.json \
  --from-file=service.key=eden-ha-install/service.key

The context's service_credential_file must be /run/eden-ha-license/private/service.key, matching the chart's private staged file. Keep its deployment UUID consistent with licensing.clusterUid and every HA TOML configuration. The signed licensed node identity stays stable across replicas; each pod retains its distinct physical identity. A browser/CLI image download session is a separate credential and does not supply this Service key.

4. Install identities and configure networking

The release identity bundle must contain these files for every ordinal. The cache-sync Secrets are needed only for a separately qualified profile that enables cache sync; the active-passive overlay above disables it:

Create those Secrets from the corresponding private bundle directories with kubectl -n eden create secret generic --from-file=. The chart stages only the current pod's private material as regular mode-0600 files in memory. Enable Kubernetes Secret encryption at rest and restrict RBAC.

Each TOML configuration must have the same deployment UUID, network identifiers, genesis committees and configuration generation across pods. Its local public keys, certificate paths, private-key paths and advertised relay address are pod-specific. Cache-sync certificate SANs must match the actual stable pod DNS names. Keep configuration generations monotonic; the runtime rejects rollback. The bundle is pre-provisioned material, not generated by Helm. Ask Eden to regenerate it if the namespace, release name, replica count or relay addresses change. The production configuration uses these default paths:

Keep each control/network Blossom, lifecycle and ordered-store directory distinct and persistent. The init container prepares the chart's directory layout. Runtime validation rejects mismatched keys, shared/temporary stores, invalid identities and configuration rollback; Helm cannot inspect those runtime invariants.

Permit only the required network paths:

Allow cluster DNS and any configured exporter endpoints. Keep database and peer ports private. The production example creates a pod-pinned UDP LoadBalancer Service per ordinal with externalTrafficPolicy: Local. Reserve stable relay addresses before generating the advertised service records; use the provider's supported Service annotations/address reservation and restrict allowed source ranges. Verify that your Kubernetes provider supports this UDP topology. Do not assume an HTTP ingress routes QUIC/UDP.

5. Render, install and verify

helm lint "$EDEN_CHART_PATH" -f eden-ha-install/values.yaml \
  -f eden-ha-install/active-passive.yaml
helm template eden-ha "$EDEN_CHART_PATH" --namespace eden \
  -f eden-ha-install/values.yaml -f eden-ha-install/active-passive.yaml \
  >eden-ha-install/rendered.yaml

Review the immutable image, pod names, Secret references, topology constraints, one PVC per pod, and TCP/UDP Services. A successful render checks chart values; it does not validate certificate contents, license eligibility or network reachability. Run the following installation commands only after the HA licensing gate has been resolved in the matched release supplied by Eden:

helm upgrade --install eden-ha "$EDEN_CHART_PATH" --namespace eden \
  -f eden-ha-install/values.yaml -f eden-ha-install/active-passive.yaml \
  --wait --timeout 15m
kubectl -n eden rollout status statefulset/eden-ha --timeout=15m
kubectl -n eden get pods,pvc,services
kubectl -n eden port-forward pod/eden-ha-0 8000:8000

In another terminal, check each path, then repeat the port forward for every pod:

for path in health ready health/cache-sync health/consensus-control health/nat-relay; do
  curl --fail --show-error "http://127.0.0.1:8000/$path"
done

/health checks process liveness; /ready gates serving. The other endpoints check peer catch-up, consensus projection and relay membership. A live process alone does not establish that the HA fleet is ready. Investigate rejected identities, missing Secrets, database access, required network leases, peer DNS and blocked ports before routing customer traffic.

Finish acceptance with authenticated endpoint registration and a disposable write/read through the actual gateway protocol. Set each native listener port in interlayPorts, ensure it matches the listener created through Eden, and allow the selected client-to-listener and Eden-to-backend paths. In a non-production rehearsal, replace one pod at a time, confirm private state is retained and peers catch up, and verify client reconnect/retry behavior. Check the quorum table before expecting control-plane mutations to continue during that test. Record the actual release digest, identities and results.

6. Operate and upgrade

Keep the same tier and deployment profile for a routine image update. Existing single-instance Professional or Enterprise customers need the corresponding qualified single-instance image. Moving an embedded deployment to external PostgreSQL/ClickHouse and multiple replicas is a separate planned datastore transition; replacing its image alone does not perform that transition.

Keep PostgreSQL/ClickHouse backups, per-pod coordination state, encryption keys, identity Secrets, licensed deployment identity and Helm values together in the recovery plan. Persistent outboxes hold accepted work; deleting a pod's PVC is not a recovery step. Follow backup and restore and upgrade and rollback.

Use a matched, qualified HA image/chart for an upgrade. Preserve license and deployment identities and roll one ordinal at a time only while the required quorum/readiness checks remain healthy. In a three-validator configuration, plan for consensus-gated changes to pause while one validator is unavailable. Do not detach licenses merely to change image bytes. Confirm schema rollback compatibility before using helm rollback; reverting the image does not undo database changes or configuration-generation watermarks.

The active-passive overlay uses OnDelete updates. Stop and fence the old ordinal before replacing it; preserve its UUID, keys, configuration, license custody and PVC. A replacement must have a schedulable host compatible with that volume's zone and attachment policy. The other four validator hosts are not spare capacity under the required hostname anti-affinity.

A crash-interrupted migration remains fenced and may require restoring safe source routing, explicit target reconciliation and a fresh operation. Do not restart it blindly or delete its audit history. Follow migration operation recovery. Customer configuration determines failure detection, fencing, storage reattachment, scheduling, readiness and traffic-restoration time. No fixed recovery duration or live AKS storage/load-balancer qualification is implied.

For observability, use monitoring and telemetry exporters. Datadog or an OTLP collector can receive exported telemetry; the current HA chart still requires ClickHouse for its durable analytics/outbox path. Exporter selection does not replace that datastore.