Troubleshooting

Symptom → fix, condensed from §16 — Troubleshooting. The Engine degrades quietly more often than it crashes loudly — when behavior is wrong but nothing is red, trust the /api/health/detailed body and the bracketed journal tags ([VRAM], [MODEL-ROUTING]) over the absence of an exception.

Gather signal first

1. GET /api/health (API + Postgres) and GET /api/health/detailed — read the body.
2. journalctl -u aetherium.service -n 200 --no-pager | grep -iE "error|exception"
3. docker ps — Postgres + ChromaDB up?

Escalating? Include the deployed SHA (cat /opt/aetherium/deployed-commit.txt), the /api/health/detailed body, and the journal lines around the failure. For request forensics: AETHERIUM_HTTP_VERBOSE=1 in the service env, restart (loud — turn back off).

Build-time

Symptom	Cause	Fix	See
Publish "works" but client 404s	Published the .slnx, not the server .csproj — no wwwroot/	Publish the server .csproj	§14
NETSDK1045: …does not support targeting .NET 10.0	No .NET 10 SDK on host	Install the .NET 10 SDK	§14
deploy.sh: working tree has staged/modified changes	Refuses a dirty tree	Commit/stash, or --force	§14
deploy.sh: bun not found (exit 7)	The client build needs bun	Install bun, or --no-spa	§14

Deploy-time (Linux · systemd)

Symptom	Cause	Fix	See
Service restarts forever; 203/EXEC or libhostfxr.so could not be found	DOTNET_ROOT/ExecStart wrong	Point both at the real .NET install; daemon-reload; restart	§14
Loads none of your config; REPLACE_ME… in logs; storage falls back	WorkingDirectory ≠ install dir	WorkingDirectory=/opt/aetherium in the unit	§14
appsettings.Linux.json silently ignored	ASPNETCORE_ENVIRONMENT ≠ Linux (loads by name, not OS)	Environment=ASPNETCORE_ENVIRONMENT=Linux in the unit	§09
Database migration failed, later calls 500	Postgres not ready at startup; exception swallowed	Start Docker, confirm Postgres, restart	§14
deploy did not become healthy within 60s	Service up but not answering /api/health	Run the printed journalctl cmd; work the rows above	§15

First-start & configuration

Symptom	Cause	Fix	See
Won't start: [ToolCatalogValidator] Tool catalog drift detected	Router catalog ≠ prompt docs; drift fatal by default	Give each tool a router mapping and a ToolDoc; bisect with Aetherium:ToolCatalog:FailOnDrift=false	§12
An env var is silently ignored	appsettings.Local.json outranks env vars	Set it in Local.json, or remove it there	§09
A model is never used; a different one runs	Its ID has a cloud prefix (gpt-, claude-, …) — skipped	Use a real local ID, or leave blank/default	§09
A changed setting has no effect	Inert (pool keys) or shadowed (Aetherium:Cleanup timers)	Change cleanup timers as SystemSettings rows; pool keys do nothing	§09
Exits: Kestrel address already in use (:5257)	Prior instance / port held	Stop the other process, restart	§02

Runtime

Symptom	Cause	Fix	See
ChromaDB 500: unable to open database file (SQLite 14)	Volume on NTFS (ignores POSIX locks) or FD exhaustion	Keep volume on ext4/btrfs/xfs + nofile: 65536 ulimit	§14
Error: Unknown tool '<name>'	No dispatch arm in McpToolRouter.ExecuteToolAsync	Add the switch arm	§12
New tool runs with no approval, truncated audit	Absent from ToolSecurityLevels → Standard	Add it at Elevated/Restricted	§07
Disk fills; ToolSecurityAuditEntries unbounded	No retention exists	Schedule a periodic DELETE by Timestamp	§15
/api/health 200 while inference/disk broken	Basic check is Postgres-only	Parse /api/health/detailed body	§15
Agents hang then LM Studio API Error; empty model picker	LM Studio unreachable / no model loaded (10-min timeout)	Start LM Studio, load a model, verify …:1234/v1 and /v1/models	§13
Container initialization error… Docker may not be running	Docker unreachable / user not in docker group	Start Docker; on Linux add the service user to docker	§14
Error: Domain '<host>' is not in the allowed domains list	Team allowed_domains set, host not on it	Add the domain to the template	§07
Outbound data blocked — potential credential leak…	DLP matched a secret in a tool arg	Remove the value; pattern lives in EgressPolicyService	§07
VRAM orchestration failed… / [VRAM] ABORT	The run's models will not fit beside what must stay resident (another lease's models or a foreign model; the standard set is evicted, not protected); the message carries the numbers and the holder	Smaller model/quant, move that model's KV cache to system RAM via the sidecar's load_config, record a measured ModelCosts entry, or wait for / release the holding lease (GET /api/inference/queue)	§09
Agent stops mid-run; Circuit breaker: <reason>	A safety ceiling tripped (not a crash)	Usually correct; widen the team's circuit-breaker limits if too tight	§06
search_knowledge empty though KB populated; ChromaDB… marking unavailable	ChromaDB down → retrieval degrades to empty	Bring ChromaDB back (docker ps); auto-recovers	§15
No model available on any instance, failing <N> tasks	No instance (target or alternate slot) reachable with the model	Confirm ≥1 healthy instance with a loadable model	§13

Integration

Symptom	Cause	Fix	See
Long build → 502/504 from your proxy, Engine fine	run endpoints synchronous; proxy read timeout severs it	Raise proxy read timeout to longest build	§13
Webhook retries never arrive after a restart	Dead-letter queue is in-memory, cleared on restart	Reconcile must-not-miss events against the runs API	§13
Secondary LM Studio always unhealthy	Secondary points at a LAN address dead on a single box	Disable Secondary, or repoint to localhost	§13
Model loads move but chat hits the old endpoint	Two config homes: chat reads AI_BaseUrl, mgmt reads appsettings	Set the chat endpoint in the Settings page	§09