Eden metrics, traces, and logs

Use this guide with exporter setup, the metric semantics and dashboards, and the API discovery guide.

Complete source catalogs

The generated catalogs enumerate first-party production declarations and emission sites, with source links so each record can be checked:

This is a source inventory across Eden product components, not a promise that every entry appears at /metrics or in every running deployment. Orchestrator, Licensing, LLM Hosting, and other executables expose their own signals; the source component identifies them. Test-only items, test files, examples, benchmarks, tools, and vendored dependency instrumentation are excluded. Dependencies can emit additional diagnostics. Attached cfgs are recorded, but module/build gates and runtime selection still determine which sites are reachable. Macro-generated and runtime-selected names are documented as their producer or expression; they are not an enumerable set of live IDs.

Metrics: names, types, units and labels

The Service's AllMetrics registry contains these fast metric groups:

Each generated metric row links to its declaration and exact help text. The curated metrics reference documents units, collection points, labels, migration timing semantics, and dashboard interpretation by group. Recording methods determine each instrument's actual labels; a shared label schema does not mean every label is attached to every series.

For derived instruments, DogStatsD uses .. The derive passes _ to Prometheus, OTLP and ClickHouse exporters. For example, gateway.request_duration_microseconds becomes gateway_request_duration_microseconds on a direct scrape. Keep punctuation in prefixes in mind: the current direct text writer preserves dots in a prefix (for example, eden.endpoint_active_requests); it does not replace them with underscores. Inspect actual exporter output and your receiver's accepted naming format when moving queries between backends. Handwritten comparison and semantic exporters have their own literal names. The JSON catalog records the declaration and names separately; it does not normalize Collector-added namespaces or backend-specific suffixes.

Most core/gateway/endpoint durations are microseconds; semantic timing histograms have their own units, and retained trace duration is nanoseconds. Use each instrument's documentation and recording conversion instead of inferring units solely from a suffix. Do not turn histogram bucket values into unweighted observations. Do not average percentiles across nodes or sum cumulative snapshots over time as though they were deltas.

Native OTLP distributions use cumulative exponential histograms (scale 0, base-2 buckets), while the direct Prometheus text form keeps only count and sum. A Collector translating the OTLP path must preserve histogram counts and bounds; a text-summary scrape cannot reconstruct the discarded distribution.

Dynamic families are created when traffic records a label set and may be evicted after inactivity. Missing series can mean idle, disabled, evicted, unavailable or lost data; absence does not establish zero. Exporter queues and span collectors are bounded. Export loss counters, provider state and query completeness must accompany evidence used for a migration gate.

Common request dimensions originate in TelemetryLabels: eden_node_uuid, HTTP method/path/status, gRPC method, org_uuid, user ID/UUID, endpoint ID/UUID/kind, request/access kind, request/correlation IDs, client context, and traffic_class. internal marks control-plane work; external marks customer traffic. Retained telemetry rows use organization_uuid; shared source instruments commonly use org_uuid. _system is a system sentinel, not a customer organization. Check label presence before applying a traffic filter to older/unlabeled data.

Prometheus is deployment-wide and its bearer token does not filter tenants. Organization-scoped API reads derive organization identity from authentication. Avoid promoting user IDs, paths, client addresses or arbitrary error strings to unbounded downstream tags.

Traces: producers and record fields

Eden has two trace producers:

1. Native fast spans: the HTTP middleware produces eden.http.request server

spans; #[with_telemetry] produces a handler-named server span and a tracing info span. Explicit client_tracer/server_tracer calls describe authentication, database/cache operations, endpoint dispatch, protocol and migration work. The generated trace catalog includes every recognized producer and add_event site with arguments.

1. tracing instrumentation: #[instrument] and tracing::*_span! create

spans routed through the OpenTelemetry SDK when OTLP is enabled. These use gRPC export. Native spans use a separate HTTP/ClickHouse fanout path.

W3C traceparent/tracestate propagate request ancestry. Responses can include traceparent; logs carry trace_id and span_id for correlation. Native HTTP spans include component=http, http.method, http.route, status, request labels and authenticated organization scope when established. Handler events add their own attributes; inspect the emission site for the exact event schema.

The retained native trace row is defined by TraceRow:

Native producers buffer spans, with a 64-span publication target and bounded per-thread queues. HTTP completion handling flushes completed request work; other producers and idle threads can have different visibility timing. Sampling, producer drops, fanout capacity and export failure can make a trace partial. The collector's DogStatsD diagnostics are span.sampling.recorded, span.sampling.sampled_out, and span.collection.dropped; these do not imply all three are exposed by every other exporter. Enabling log level trace is not the same as enabling distributed trace export.

Logs: producers, severities, audiences and fields

Structured eden_logger_internal::log_* records are sent to an installed bounded sink callback and independently formatted to the console according to the writer. log/tracing diagnostics and println!/eprintln! are also listed in the source catalog, but they do not all pass through the structured Eden callback. Use a container agent to retain those streams.

Structured severity is TRACE, DEBUG, INFO, WARN or ERROR. Audience is Internal, Client or Both in the typed enum (display/rendering can use uppercase). Audience is separate from severity and from authorization: the telemetry query API applies its authenticated tenant/plane fence and permissions, while audience is a separately filterable field. Do not treat the audience enum alone as a tenant access control.

The structured schema combines EdenLog and EdenRequestFields:

The retained LogRow adds service_name, node_uuid, telemetry_plane, data_class=operational_log, and a labels string-pair map. Optional context can become empty columns; do not assume a background/system event has a customer principal. Console display text, the typed structured record and the retained analytics row are different representations of the same producer where that producer uses the callback.

Frequent log families include startup/configuration, auth/RBAC decisions, endpoint/backend failures, gateway connection termination with reason and queue snapshots, migration phase/recovery/cutover, workflow execution, telemetry queue/export outcomes and managed-control/VPN lifecycle. The generated log catalog supplies the actual message templates and fields for every recognized first-party emission site rather than a fixed list of possible fully formatted messages.

Audit and analytics event records

These are separate typed records, not extra Prometheus series or a guarantee that the event appears in the operational log table:

See audit exporter setup for ClickHouse/S3 fanout. Audit rows and request-analysis fields can contain customer identifiers/key patterns: apply your retention and access policy. Verify credential redaction on the exact image before enabling request analysis or sharing logs. Keep AUTH arguments and request bodies out of downstream metric tags and diagnostic bundles; apply bounded retention and access controls to request-derived telemetry.

Read retained signals

Use an organization JWT with ControlPerms::READ, not the deployment's Prometheus bearer. The canonical API supports query- or path-selected signals:

curl --fail --silent --show-error \
  --header "Authorization: Bearer $EDEN_TOKEN" \
  "$EDEN_ORIGIN/api/v1/analytics/telemetry/metrics?range=1h&limit=100"
curl --fail --silent --show-error \
  --header "Authorization: Bearer $EDEN_TOKEN" \
  "$EDEN_ORIGIN/api/v1/analytics/telemetry/traces?range=1h&limit=100"
curl --fail --silent --show-error \
  --header "Authorization: Bearer $EDEN_TOKEN" \
  "$EDEN_ORIGIN/api/v1/analytics/telemetry/logs?range=1h&limit=100"

GET /analytics/telemetry?signal=metrics|traces|logs is equivalent. Common parameters are inclusive RFC 3339 from/to, lookback range, order, limit and offset. Default limit is 500, maximum 5,000, maximum offset 100,000, default range one hour, maximum range 365 days; provider capacity or retention can impose tighter bounds. Use explicit time bounds when paging a moving dataset. Signal-specific filtering follows the handler's parameter validation. The /analytics/clickhouse paths are compatibility aliases; new clients should use /analytics/telemetry.

For ShardTelemetry-capable builds, GET /api/v1/analytics/telemetry/status?signal=traces reports authenticated provider observations. Interpret unsupported, unavailable, stale, incomplete, capacity-exhausted and tenant-mismatch evidence explicitly. An empty result or an installed adapter is not proof of complete evidence. Endpoint audit history uses /api/v1/endpoints/{endpoint}/analytics/audit-trail with its own keyset pagination and permissions; migration test-run logs and scheduled-agent logs also have dedicated APIs. Find them in the full API reference.

Refresh and verify the catalogs

The generator parses Rust syntax rather than matching strings in comments or test fixtures. It does not build or execute the service and does not expand arbitrary Rust macros. Its supported producer forms and explicit handwritten export adapters are in the script; extend them when a new instrumentation API is introduced. The live exporter output remains authoritative for a deployed binary, and the source catalogs are an audit aid for the version in this tree.

python3 -m venv /tmp/eden-doc-reference
/tmp/eden-doc-reference/bin/pip install tree-sitter==0.25.2 tree-sitter-rust==0.24.0
/tmp/eden-doc-reference/bin/python scripts/generate-telemetry-reference.py
/tmp/eden-doc-reference/bin/python scripts/generate-telemetry-reference.py --check
/tmp/eden-doc-reference/bin/python -m unittest discover \
  -s scripts/tests -p test_telemetry_reference.py

--check compares all generated artifacts and fails when they are stale. Review newly added producer forms and handwritten exporters during a release; there is no static generator that can prove which runtime-selected names or third-party diagnostics a workload will emit.

Search the JSON without loading the large tables in a browser:

jq -r '.metrics[] | select(.name | test("redis|migration"; "i")) |
  [.name, .export_name, .kind, .source, .line] | @tsv' \
  docs/reference/telemetry-catalog.json
jq -r '.logs[] | select(.level == "ERROR") |
  [.source, .line, .message] | @tsv' docs/reference/telemetry-catalog.json