Skip to content

16 — Troubleshooting

Find your symptom, understand the cause, apply the fix.

This is the diagnostic reference for The Aetherium Engine: a symptom-to-cause-to-fix index for the failures you are most likely to hit, ordered by when you hit them — building, deploying, first start, runtime, and at the integration edges. Each entry carries the error signature you would actually see and a link to the section that tells the full story. It is built for fast lookup: scan for your error text, don't read it front to back. Every entry follows the same shape — the symptom and its signature, the cause, the fix, and the section that carries the full story.

Most of the commands below are run on the deployed Linux host — journalctl, docker, curl, psql — because deployment is a Linux concern (§14 — Deployment). The build-time fixes are the exception, and they call out the platform that hits them.

Gather signal first

When something breaks, four moves tell you where to look before you start changing things. All four are detailed in §15 — Operations; this is the short version.

  • Hit the health endpoints. GET /api/health tells you only whether the API is up and PostgreSQL is reachable. GET /api/health/detailed reports the database, Docker, each LM Studio instance, the scheduler, active builds, and disk — but read its body, not just its status code (see the false-green entry under Runtime).
  • Read the journal. The Engine runs as aetherium.service, so journalctl -u aetherium.service -n 200 --no-pager is your log; pipe it through grep -iE "error|exception" to cut to the failures.
  • Check the containers. docker ps — are PostgreSQL and ChromaDB both up? A surprising share of "the Engine is broken" reports are a stopped container.
  • Probe the model. If an agent is misbehaving rather than the platform failing, run the benchmark harness to confirm the model still drives tools end to end (§11 — Benchmarking).

Build-time failures

The failures you hit before the Engine ever runs — compile errors, the wrong publish artifact, and the deploy script's pre-flight checks.

Symptom Cause Fix
Publish "succeeds" but the deployed Engine serves the API and returns 404 for the browser client. You published the .slnx solution (or copied a dotnet build output) — neither assembles wwwroot/. Cross-platform. Publish the server .csproj directly. §10 · §14
Build or publish on the host fails: error NETSDK1045: The current .NET SDK does not support targeting .NET 10.0. No .NET 10 SDK on the host — the deploy script publishes on the host, so it needs the SDK, not just the runtime. Install the .NET 10 SDK. §14
deploy.sh stops immediately: working tree has staged/modified changes — commit, stash, or --force. The deploy script refuses a dirty tree (untracked files are ignored). Commit or stash your changes, or pass --force. §14

Deploy-time failures

Between a successful publish and a running service. These are all Linux/systemd.

Symptom Cause Fix
The service restarts forever. The journal shows status=203/EXEC, or a libhostfxr.so could not be found / "You must install or update .NET" message, ending in aetherium.service: Start request repeated too quickly. DOTNET_ROOT or the ExecStart path don't point at the real .NET install. Point both at the actual install, then systemctl daemon-reload and restart. §14
The Engine starts but loads none of your config — REPLACE_ME_VIA_LOCAL_OVERRIDE shows in logs and storage falls back to /var/aetherium/… defaults. WorkingDirectory isn't the install dir; ASP.NET Core reads appsettings*.json from the working directory. Set WorkingDirectory=/opt/aetherium in the unit. §14
appsettings.Linux.json values silently don't apply (storage paths, ChromaDB endpoint). ASPNETCORE_ENVIRONMENT isn't literally Linux — the overlay loads by name, not by OS. Set Environment=ASPNETCORE_ENVIRONMENT=Linux in the unit. §09 · §14
Startup logs Database migration failed, the Engine keeps running, and later data calls return 500. PostgreSQL wasn't ready at startup — the migration exception is logged and swallowed, so the Engine runs against an unmigrated schema. Start Docker, confirm Postgres accepts connections, then restart the service. §14 · §15
deploy.sh aborts before it publishes: deploy dir not writable by <user>: /opt/aetherium. The install directory isn't owned by the service user the deploy runs as. chown /opt/aetherium to that user, so the script can sync into it without sudo. §14
deploy.sh stops before it publishes: bun not found on PATH, $BUN_INSTALL/bin or ~/.bun/bin (exit status 7). The React client build needs bun, and the user the deploy runs as has none in those places — a non-login shell often lacks ~/.bun/bin on its PATH. Nothing was changed. Install bun for that user, or pass --no-spa to deploy the server and keep the previously built client. §14
deploy.sh stops after publishing with render-gpu-env: refusing to pin: and the engine config cannot pin every GPU unit (exit status 6). The GPU pin preflight refused the config about to be deployed: no slot's Unit is llama-server.service, the slot names no Card (a GPU service on such a slot is placed without a pin only when no configured card is in this machine — a box whose one GPU is integrated) or one Aetherium:Gpus:Cards lacks, the card's Uuid is not in sysfs, or the binary its build names does not exist. For a card with a Host the same check runs on that machine and ends the GPU units on <host> cannot be pinned; a card there with no ListenAddress is refused too. Nothing in /opt/aetherium was changed. Add the cards and the slot's Card/Unit to appsettings.Local.json — each refusal line names its problem — then redeploy; scripts/render-gpu-env.py --validate checks without writing. §14
deploy.sh ends red: service did not become healthy within 60s, then prints a journalctl command. The service published and restarted but isn't answering /api/health — almost always one of the failures above. Run the printed journalctl command and work the rows above. §14 · §15

That last one is usually your first signal that a deploy went wrong. The deploy script polls /api/health after restarting the service, so a health-timeout means the service came up unhealthy — a crash-loop, a config-load failure, or a database that wasn't ready. The journalctl line the script prints is where the real cause shows.

First-start and configuration failures

It won't start, or it starts but behaves wrong — the first-run traps.

Symptom Cause Fix
The Engine won't start: [ToolCatalogValidator] Tool catalog drift detected: then Missing ToolDocs entries (…) or Stale ToolDocs entries (…). The routable tool catalog and the prompt-enhancer's docs disagree; drift is fatal by default (Aetherium:ToolCatalog:FailOnDrift, default true). Give every tool both a router mapping and a ToolDoc; to bypass during a bisect, set FailOnDrift to false. §06 · §12
An environment variable you set is silently ignored. appsettings.Local.json is appended after env vars and outranks them. Set the value in appsettings.Local.json, or remove it there so the env var wins. §09
A model you configured is never used; the agent silently runs a different one. Its ID starts with a cloud prefix (gpt-, claude-, o1-, o3-, gemini-, palm-, command-), which the local-inference path skips. Use a real local model ID, or leave the field blank / default to defer to the template. §09
A setting you changed has no effect. It's inert — the Aetherium:Cleanup container timers are shadowed by SystemSettings rows of the same name. Change the cleanup timers as SystemSettings rows in the Settings page. §09 · §15
The Engine exits at startup; Kestrel reports Failed to bind to address http://localhost:5257: address already in use (or 7045). A prior instance is still running, or another process holds the port. Cross-platform. Stop the other process or free the port, then start again. §02

Runtime failures

It was working; now something's wrong.

Symptom Cause Fix
ChromaDB calls return 500: unable to open database file (SQLite code 14). The ChromaDB volume is on NTFS (whose Linux driver ignores the POSIX advisory locks SQLite needs), or file-descriptor exhaustion past the default 1024 open-file limit once collections pile up. Keep the volume on ext4/btrfs/xfs and keep the compose nofile: 65536 ulimit — both are the compose defaults. §14
An agent's tool call returns Error: Unknown tool '<name>'. The tool is mapped and visible but has no dispatch arm in McpToolRouter.ExecuteToolAsync. Add the switch arm that routes the call to your method. §12
A new tool runs with no approval gate and only a truncated audit entry. It's absent from the ToolSecurityLevels map, so it defaults to Standard — and the boot validator doesn't check this map, so nothing warns. Add the tool to the map at its intended Elevated/Restricted level. §07 · §12
Disk fills over weeks; the ToolSecurityAuditEntries table grows without bound. Every gated tool call is appended and nothing prunes it — no retention key, not even on project deletion. Schedule a periodic DELETE by Timestamp (recipe in §15). §09 · §15
/api/health returns 200 healthy while inference, disk, or the scheduler is broken. Basic health checks PostgreSQL only; the detailed endpoint flips to 503 only for the database or Docker. Parse the /api/health/detailed body for the LM Studio, disk, and scheduler fields. §15
Agents hang, then fail: LM Studio API Error: <status> - <error>; the model picker is empty (logs Error fetching models:). LM Studio at the configured base URL is unreachable or has no model loaded; a stuck model hangs to the 10-minute timeout. Start LM Studio, load a model, verify the base URL (http://localhost:1234/v1) and that /v1/models lists it. §09 · §13
A container-backed tool fails: Container initialization error: <msg>. Docker may not be running or image pull failed. The Docker daemon is unreachable — not running, wrong socket, or (Linux) the service user isn't in the docker group. Start Docker; on Linux add the service user to the docker group. §14
An agent's web/HTTP tool is refused: Error: Domain '<host>' is not in the allowed domains list. The team's allowed_domains allowlist is set and the host isn't on it. Add the domain to the template's allowed_domains. §07 · §13
An agent's outbound tool is blocked: Outbound data blocked — potential credential leak detected in '<arg>'. The DLP pass matched a secret pattern (API key, token, private key) in a tool argument and refused to send it. Remove the sensitive value from the argument; if it's a false positive, the pattern is in EgressPolicyService. §07
A run aborts before agents start: Error: VRAM orchestration failed — … (journal: [VRAM] ABORT — …). The run's models will not fit on the card beside what must stay there. The message carries the arithmetic: what was needed, the budget, and what is being held (a model another live lease holds — the reason names the holder — or one the Engine did not load and so will not evict; the standard set is evicted, not protected). Use a smaller model or quant; wait for or release the lease that holds the card (GET /api/inference/queue shows it); or unload the foreign model yourself. §09
An agent stops mid-run and looks hung; the journal shows Circuit breaker: <reason>. A safety ceiling tripped (step timeout, re-plan cap, an agent's turn budget spent) — not a crash. Usually the right behavior; if too tight for a reasoning model, widen the team's circuit-breaker limits. §06
search_knowledge returns nothing although the knowledge base is populated; the journal shows ChromaDB … marking unavailable. ChromaDB is down, so retrieval degrades to empty results instead of erroring on each call. Bring ChromaDB back (docker ps); it re-initializes on recovery (look for ChromaDB recovered). §15
A batch of agent tasks all fail at once: No model available on any instance, failing <N> tasks. Neither the target instance nor any alternate slot is reachable with the requested model, or none of them can load it. Confirm at least one instance is healthy with a loadable model; the preceding [VRAM] Load failed lines say why. §13

Notice how many of these never raise an error — the model picker just empties, a setting just does nothing, knowledge search just returns nothing. That is the runtime failure mode to internalize: the Engine degrades quietly more often than it crashes loudly. When behavior is wrong but nothing is red, trust the /api/health/detailed body and the bracketed journal tags ([VRAM], [MODEL-ROUTING]) over the absence of an exception.

Integration failures

At the edges, where the Engine meets other systems — the proxy in front, the webhooks behind, and the inference server it depends on.

Symptom Cause Fix
A long build dies with a 502/504 from your proxy while the Engine keeps working. The run endpoints are synchronous and hold the connection open for the whole build; a default 30–60s proxy read timeout severs it. Raise the proxy read timeout to your longest expected build. §13 · §14
Webhook events pending retry never arrive after a restart. The dead-letter queue is in-memory and cleared on restart. For must-not-miss events, reconcile against the runs API instead of trusting the webhook stream. §13 · §15
The Secondary LM Studio instance is always unhealthy: LM Studio Secondary (…) is now unhealthy. The shipped Secondary instance points at a LAN address; on a single-box install that address is dead. Disable Secondary, or repoint it to localhost. §09 · §13
Model loads and unloads move, but chat requests still hit the old endpoint. LM Studio has two config homes — chat/routing reads SystemSettings (AI_BaseUrl); management/VRAM reads the Aetherium:LMStudio:Instances appsettings block; they aren't synced. Set the endpoint you chat against in the Settings page. §09 · §13

Still stuck?

When your symptom isn't above, widen the net in this order. The detailed health endpoint (GET /api/health/detailed) is the fastest subsystem-by-subsystem readout — start there (§15). For request-level forensics, set AETHERIUM_HTTP_VERBOSE=1 in the service environment and restart; it raises HTTP logging to a full trace, so turn it back off when you're done — it's loud. And the section that owns the failing subsystem almost always carries the deeper explanation: the entries above link straight to it.

When you escalate, make the report worth reading: include the deployed commit SHA (cat /opt/aetherium/deployed-commit.txt), the /api/health/detailed body, and the journal lines bracketing the failure. That trio plus the symptom is usually enough to place a new problem against one of the entries here — or to prove it's a genuinely new one.

Where to go next