Skip to content

13 — Integrations

How the Engine connects to the systems around it — its inference backend, an image backend, a search backend, and the external orchestrators that drive it.

The Aetherium Engine runs locally, but it is not self-contained. It leans on a small set of external peers to do its work: an inference server for the LLM call behind every agent turn, a search backend for web research, and — optionally — an image-generation backend. It also exposes a control surface in the other direction, letting an external system drive it: start builds, poll their status, and receive events as they happen. This section is the integration view of all four. It owns the how it connects; the services that do the connecting live in §06 — Key Services, the tools that ride on top in §07 — MCP Architecture, and the keys that point at each peer in §09 — Configuration.

One clarification before the peers: the Engine's own datastores — PostgreSQL for relational state, ChromaDB for the knowledge base — are infrastructure, not integrations. They are part of the Engine's deployment, not external systems it talks to as a client, and they are covered in §03 — System Architecture and §06 — Key Services. This section is about the peers that sit outside that boundary.

graph TB
    EXT["Your external orchestrator"]
    ENG["The Aetherium Engine"]
    LM["LM Studio — inference"]
    SX["SearXNG — web search"]
    CF["ComfyUI — image generation"]

    EXT -->|"POST /api/orchestrator/run"| ENG
    ENG -.->|"signed webhook POST"| EXT
    ENG -->|"OpenAI-compat /v1 + native /api/v0"| LM
    ENG -->|"GET /search?format=json"| SX
    ENG -->|"REST submit-then-poll"| CF

The Engine and its integration peers. Inference, search, and image generation are outbound — the Engine is the client. The external orchestrator is bidirectional: it drives the Engine over HTTP and receives signed event callbacks.

LM Studio — the inference backend

Every agent turn ends in an HTTP call to an inference server, and by default that server is LM Studio. The Engine talks to it over two distinct APIs, and the split between them is the thing to understand.

The first is the OpenAI-compatible API. Inference goes to POST /v1/chat/completions — the same contract any OpenAI-compatible client speaks, which is exactly why the Engine works with any server that implements it and isn't locked to LM Studio. LMStudioService builds the request (model, messages, temperature, max_tokens, an optional tools array, response_format for JSON-schema-constrained output, and a seed when a caller pins one for determinism) and reads back the standard response: choices[0].message.content, usage token counts, and a finish_reason. Streaming uses Server-Sent Events. Model discovery is GET /v1/models. The service exposes ChatAsync, ChatDetailedAsync (which also returns token metadata), ChatWithToolsAsync, and StreamCompletionAsync — every agent turn in the Engine bottoms out in one of these.

The second is LM Studio's native management API, covering what the OpenAI contract does not: loading and unloading models. LMStudioManagementService calls GET /api/v0/models to list every model with its load state (loaded / loading / not-loaded), POST /models/load to bring one into VRAM, and POST /models/unload to evict it. Anything but a 200 from /api/v0/models is a failed probe: the Engine then treats the instance's load state as unknown — no load, no unload, no assumption that the instance is empty — rather than reading names from GET /v1/models, which carries no load state. The core operation is "make sure model X is resident": probe what the card holds, plan against its size, load only what is missing, unload only what the shortfall requires, verify — which is what keeps a multi-model agent swarm from thrashing a single GPU. The embedding model, the instance's standard set and anything the Engine did not load itself are never evicted, and a model the plan left in place beside the run's own is reported by the post-load check, not treated as a failure; a model that did not land, or a probe that did not answer, still fails the run. Loads are serialized per instance, and a load already in flight (state loading, whether started by the Engine or externally) is awaited rather than re-requested — LM Studio answers an unmergeable duplicate load request by spawning a second copy of the model under a suffixed instance id (model-key:2), silently doubling its VRAM footprint. If a stacked duplicate slips through anyway, the post-load verification detects and unloads it.

Routing across instances is a third, separate concern. §06 — Key Services covers ModelSchedulerService, the channel-based queue that groups pending work by model so the loader swaps as rarely as possible, and which also decides which instance a call is sent to — from where the model is resident, which instances answered their last probe, and any affinity pin the run set. LLMRouterService holds the instance configuration and in-flight counters that decision reads. The integration fact to internalize here is that loading and inference are addressed independently: a model is loaded against one URL and inference is sent to another. That separation produces a configuration gotcha sharp enough to earn its own treatment — changing an instance's configured URL moves where models load, not where inference is sent. That gotcha, and the keys behind it, are in §09 — Configuration.

A quick reachability check against any inference server — list the models it's serving:

Invoke-RestMethod http://localhost:1234/v1/models | ConvertTo-Json -Depth 4

curl -s http://localhost:1234/v1/models | jq '.data[].id'

ComfyUI — image generation

When an agent needs an image, it calls the comfyui_generate tool, and ComfyUIService turns that into a job on a ComfyUI backend. ComfyUI's API is REST plus a WebSocket progress stream; the Engine uses the REST half and polls — it does not consume the WebSocket.

The flow is submit-then-poll. The service POSTs a workflow graph to POST /prompt and gets back a prompt ID. It then polls GET /history/{prompt_id} every two seconds until the workflow's output images appear or a five-minute timeout elapses, and downloads each result from GET /view into the run's workspace — <save_dir>/<file>.png, images/ by default — so the path the agent gets back is one it and its teammates can read. (The submit call itself allows a generous timeout — an SDXL checkpoint cold-starting on the backend can take well over a minute before the prompt is even accepted.) The two MCP tools map directly onto the two halves of that flow: comfyui_generate submits and, by default, waits for completion before returning the image filenames, while comfyui_status reports queue depth or the state of a specific prompt ID — useful when a caller submits without waiting and wants to poll on its own schedule.

The workflow can be supplied two ways. Pass a full ComfyUI workflow graph as JSON and the service submits it verbatim. Pass plain text and the service wraps it in a built-in SDXL graph — checkpoint loader, positive and negative prompt encoders, sampler, VAE decode, save node — so a text prompt produces an image without the caller knowing ComfyUI's node format; the checkpoint that graph loads is the ComfyUI_Checkpoint setting when set, otherwise the first checkpoint the install reports through ComfyUI's object_info, and with neither the tool refuses instead of queuing a graph that cannot load (the graph used to name one fixed checkpoint file). The backend URL comes from the ComfyUI_Endpoint setting (default http://localhost:8188) which, unlike most keys, lives in the runtime settings store rather than appsettings.json. The two tools are detailed in §07 — MCP Architecture; the configuration in §09 — Configuration.

SearXNG — the search backend

Web research runs through the web_search tool, and its primary backend is SearXNG — a self-hosted metasearch engine that aggregates results from upstream engines without the per-query API keys and rate limits of a commercial search API. Self-hosting it also keeps queries on your own infrastructure, which is the same data-sovereignty reasoning that makes the whole Engine local-first. ResearchTools issues a GET against the configured instance — /search?q=<query>&format=json — and returns the titles, URLs, and snippets to the calling agent. The instance URL comes from the SearXNG_Url setting.

When SearXNG isn't configured, or a request to it fails, web_search falls back to DuckDuckGo's HTML endpoint, scraping results rather than calling an API. The fallback is deliberately degraded — it exists so research doesn't hard-fail when the self-hosted backend is down, not as a co-equal path. If both are unavailable, the tool returns an empty result rather than an error.

Every outbound call a tool makes — web_search, web_fetch, http_request, download_pdf — passes through the egress gate first. EgressPolicyService enforces two policies before any request leaves the host. A domain allowlist, configured per team, decides which destinations are reachable: localhost and RFC 1918 private ranges are always allowed (that's how the Engine reaches its own SearXNG, inference, and ChromaDB), and everything else must match an allowlisted pattern, with *.example.com-style wildcards supported. A credential scan then inspects the outbound URL, query string, headers, and body for secret-shaped strings — API keys, cloud access keys, source-host tokens, private-key headers, JWTs — and blocks the request when it finds one, so an agent can't exfiltrate a credential it stumbled across in the workspace. Either kind of denial is logged with its reason, so blocked egress is observable rather than silent. That gate is the reason third-party reach is safe to grant at all; its full mechanics and the allowlist configuration are in §07 — MCP Architecture and §09 — Configuration.

Driving the Engine from outside — the inbound API

The integrations above are ones the Engine reaches out to. This one runs the other way: an external orchestrator — your own automation, a scheduler, a higher-level agent — driving the Engine over HTTP. This is the surface to reach for when Aetherium is one stage in a larger pipeline rather than something a human steers through the UI.

The control surface is the orchestrator API under /api/orchestrator. The two endpoints that matter most for external drive are:

  • POST /api/orchestrator/run — send a message to an existing project's agent team and get the response back. The body carries projectId and message, plus optional toolDefaults (pin tool arguments the agents shouldn't be trusted to set), samplingOverrides (e.g. a per-role seed, which with temperature: 0 pins output for models that honor it), handoffChecks (post-build expectations the caller attaches), and sourceAgent (who is calling; recorded on the run and in its build.completed / build.failed webhooks — absent means the Studio, ui).
  • POST /api/orchestrator/run-from-template — the atomic one: create a project from a template, scaffold its workspace, and start the build in a single call. This is usually the right entry point for an external driver, because it doesn't require a project to exist first.

On receipt, the Engine records container activity to keep the project's sandbox warm, hands the message to the project's agent team, and runs the orchestration — the same supervisor-and-workers loop a human would trigger from the UI — before it responds. Nothing about the inbound path is a side door; it drives the identical machinery.

{
  "templateId": "fullstack-web",
  "projectName": "checkout-service",
  "description": "Payment checkout microservice",
  "message": "Build a REST API for cart checkout.",
  "workspaceTier": "Production",
  "sourceAgent": "ci-pipeline"
}

The call returns the new project's ID, the build's run GUID for follow-up polling, and a status:

{
  "success": true,
  "projectId": 42,
  "runGuid": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
  "status": "completed",
  "message": "Project created and build completed.",
  "response": "..."
}

$body = @{
    templateId  = "fullstack-web"
    projectName = "checkout-service"
    message     = "Build a REST API for cart checkout."
} | ConvertTo-Json

Invoke-RestMethod -Method Post `
    -Uri http://localhost:5257/api/orchestrator/run-from-template `
    -ContentType application/json -Body $body

curl -s -X POST http://localhost:5257/api/orchestrator/run-from-template \
  -H "Content-Type: application/json" \
  -d '{
    "templateId": "fullstack-web",
    "projectName": "checkout-service",
    "message": "Build a REST API for cart checkout."
  }'

An optional overrides object on the same body (modelId, nodeModels, preferredInstance, temperature, stepRounds, toolRequests) pins the run's model, instance, sampling, step rounds or tool-request policy without editing the template; the Engine applies it to the team before the run and logs each field (API · TemplateController).

Both run endpoints are synchronous — they hold the connection until the agent team finishes and return the result inline. For a multi-agent build that can mean a long-held request, so set a generous client timeout. The research endpoints take the opposite approach: run-research returns immediately with a projectId, and you poll for progress. That polling is the same API the caller uses to follow any build: GET /api/orchestrator/runs/{projectId}/{runGuid} returns run status enriched with container and served-port details for validation, GET /api/orchestrator/runs/{projectId}/logs returns the per-step build log, and POST /api/orchestrator/runs/{projectId}/cancel stops an in-flight run.

Protect the inbound API at the network boundary

These endpoints carry no built-in authentication — they trust the caller. The Engine is designed to run inside a trusted network boundary, and the inbound API assumes that boundary exists. If you expose it beyond the local host, put it behind a layer that enforces identity — a reverse proxy with mTLS, an API gateway, a bearer-token check at the edge — or keep it on a private network. Do not expose the orchestrator API directly to the public internet. Authenticity on the outbound side is handled with a signature (below); the inbound side leaves that to your deployment.

Pushing Engine events outward — webhooks

The inbound API is pull: the caller asks for status. For push, the Engine emits webhooks as a build progresses, so an external listener can react in real time instead of polling. WebhookService owns this direction.

A listener is registered as a target: a URL, a shared secret, and the set of event patterns it subscribes to. Targets come from two places — the Aetherium:Bridge:Webhooks:Targets configuration section (static) and a database table managed through the API (dynamic), merged and cached briefly so a busy build doesn't hit the database per event. Subscriptions are glob patterns, so build.* matches every build-lifecycle event: build.started, build.step_completed, build.completed, build.failed, build.cancelled.

"Aetherium": {
  "Bridge": {
    "Webhooks": {
      "Targets": [
        {
          "Name": "ci-listener",
          "Url": "https://your-orchestrator.example/hooks",
          "Secret": "REPLACE_ME_VIA_LOCAL_OVERRIDE",
          "Events": "[\"build.*\"]",
          "Enabled": true
        }
      ]
    }
  }
}

A target whose Secret is still the REPLACE_ME_VIA_LOCAL_OVERRIDE placeholder is skipped with one warning, whatever Enabled says — it cannot authenticate, so retrying it forever would only fill the journal. The shipped appsettings.json target is disabled for that reason; set a real secret in appsettings.Local.json and enable it there.

Each delivery is a POST of a JSON event to the target URL:

{
  "event": "build.completed",
  "runId": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
  "sourceAgent": "ci-pipeline",
  "timestamp": "2026-05-29T18:22:04Z",
  "verbosity": 1,
  "data": {
    "projectId": 42,
    "elapsedSeconds": 137
  }
}

A build.completed event for a run that finished Completed but needs review also carries data.reviewNotes: the reasons, one per line (a step that completed although its verifier failed it or could not decide, a file a team node owes that is not on disk at the end, a page whose links miss a file or name one that does not exist, a page short of the pictures it owes). The same text is the run's reviewNotes in the runs API.

Operator notices. The events that need a person — finishing.awaiting_answer (a finishing brief is waiting for its answer), finishing.failed, finishing.done and tool.approval.requested — carry a top-level message that reads as it stands and ends with a link to the app page when Aetherium:Bridge:AppUrl is set, so a chat gateway can relay every one of them with a single template (an OpenClaw hook mapping whose messageTemplate is {{message}}, for example). Their body is flat — event, message, link and the ids — not the build-event shape above. Subscribe a chat target to these and leave build.* off unless every run should reach a person. A target with "Kind": "telegram" and a ChatId sends the message itself through Telegram's Bot API, its Secret being the bot token: nothing between the Engine and the chat has to run a model, which matters because the notices fire while the Engine's own models are busy.

The delivery is signed. The Engine computes an HMAC-SHA256 of the payload body using the target's secret and sends it in an X-Aetherium-Signature: sha256=<hex> header; the listener recomputes the same HMAC to confirm the event genuinely came from the Engine and wasn't tampered with in flight. (A shared-secret token header is sent alongside it for gateways that prefer a plain token check.) Two more headers ride along: X-Aetherium-Event names the event type, and X-Aetherium-Delivery carries a unique ID per attempt so a listener can deduplicate retries.

Delivery is resilient. Emitting an event is fire-and-forget for the caller — it lands on an in-process queue and a background loop delivers it. The queue is bounded; under sustained backpressure it sheds the oldest undelivered events rather than growing without limit, so a listener should treat the stream as best-effort and reconcile against the runs API for anything it must not miss. A failed delivery (any non-2xx response, or a connection error) is retried three times with exponential backoff: 2s, 4s, 8s. If all three fail, the delivery drops into a dead-letter queue that retries it every five minutes, up to five more times, before giving up. Every attempt is recorded in a WebhookDelivery audit row. The dead-letter queue lives in memory, so an Engine restart clears anything still pending in it.

What's not built in

Notice what isn't on the list: the Engine does not consume third-party MCP servers. There is no configuration to point it at an external MCP endpoint or spawn one as a subprocess, and that's deliberate — the Engine's security gates (the egress and credential checks above, plus the approval and audit layers) only apply to tools running in-process. A tool reached over an external MCP connection would sidestep all of them. The design and its rationale are in §07 — MCP Architecture, recorded as a decision in §17 — Architecture Decisions.

So integration with an external system takes one of two shapes. If the system should drive the Engine, it uses the inbound API and webhooks above. If the Engine should call the system as part of an agent's work, you add it as an MCP server class hosted by the Engine itself — a few methods and a registration, walked through in §12 — Extending the Engine. Either way, the integration stays inside the Engine's trust boundary.

Where to go next