Ai
Inference
Inference uses Eden to place and operate models across local machines, private servers, cloud infrastructure, and managed capacity.
Runtime and Fleet coordinates the runtime fleet. Gateway exposes eligible models through a governed gateway so applications do not need to know which node or provider handled a request.
Operating flow
- Enroll an eligible runtime or configure a model provider.
- Record model, runtime, hardware, and compatibility capabilities.
- Apply organization, project, identity, licensing, and placement policy.
- Route the request to an eligible provider or runtime node.
- Stream the result through the gateway.
- Record routing, latency, usage, cost, and evidence.
Runtime choices
Eden can coordinate:
- Local model runtimes on owner-controlled machines
- Private vLLM deployments
- Cloud GPU instances
- Managed model providers
- Eden-managed open-source capacity where available
Unknown or incompatible runtimes are not eligible routing targets. Matching a model name alone is not sufficient; runtime, model, hardware, licensing, security, and cache compatibility must also match.
Shared model state
Compatible workloads may reuse permission-scoped model state. Runtime and Fleet can keep state near the inference process and prepare likely-needed state before a later request arrives. Tenant, project, identity, model, runtime, and compatibility boundaries are part of every reuse decision.