Skip to content

15 — Operations

Keeping a deployed Engine running: what to watch, what grows, what to back up, and how to recover.

§14 — Deployment got The Aetherium Engine onto a Linux host and behind a boundary. This section is everything after: the day-to-day of keeping it healthy, reading what it is doing, managing what it stores, and recovering when you need to. You will read it both ways — proactively, setting up monitoring and backups, and reactively, when something looks wrong.

The Engine manages part of its own lifecycle and leaves the rest to you. It sweeps idle containers, ages out expired workspaces, prunes old archives, applies its own database migrations on startup, and runs a set of background health probes — all without intervention. What it does not do is back up your data, prune its audit trail, or defend its own API. This section covers both halves: the self-managing parts you monitor, and the operator-owned parts you have to set up.

flowchart LR
    subgraph ENGINE["The Engine manages it — you monitor"]
        direction TB
        M1["Auto-applies migrations at startup"]
        M2["Container cleanup: idle, stop, destroy"]
        M3["Workspace-tier and archive retention"]
        M4["Background health probes, log ingestion"]
    end
    subgraph YOU["You own it — you set up"]
        direction TB
        O1["Backups: PostgreSQL, ChromaDB, workspaces"]
        O2["Audit-table pruning"]
        O3["Auth boundary, secret rotation"]
        O4["Rollback procedure"]
    end

    classDef owned fill:#B87333,stroke:#2D2D2D,stroke-width:2px,color:#FFFFFF
    class O1,O2,O3,O4 owned

What the Engine handles on its own versus what you own. Monitoring tells you the left side is working; the right side only happens if you set it up. Operations is a Linux concern — the commands below are bash, journalctl, docker, and psql, run on the deployed host.

The health surface

Two endpoints report on the running Engine, and they answer different questions.

GET /api/health is the liveness probe. It checks exactly one thing: whether the Engine can reach PostgreSQL, through EF Core's CanConnectAsync. A reachable database returns 200 with a small JSON body; an unreachable one returns 503 and flips status to degraded.

curl -s http://localhost:5257/api/health

{ "status": "healthy", "timestamp": "2026-05-29T18:22:04Z", "version": "1.0.0", "database": "connected" }

Read that for what it is: a database-connectivity check and nothing more. A 200 here tells you the API is up and Postgres answers — it says nothing about ChromaDB, the inference server, Docker, or whether any model is loaded. Wire it to your uptime monitor as the "is the process alive and is its database reachable" signal, the same check the deploy script polls and the same one §02 — Getting Started uses on a dev box.

For a real dependency check, use GET /api/health/detailed. It aggregates six subsystems — the database, Docker (with active and errored container counts), each configured LM Studio instance (reachability and the model currently loaded), the model scheduler's pending-task count, the active build count, and disk (free space on the workspace drive, temp-folder size) — plus two retrieval blocks: embedding (whether a canary embed answers) and rag (whether the active knowledge generation is in the embedding space the Engine queries in; degraded names the collections that are not). One caveat to internalize before you page on it: the detailed endpoint returns 503 only when the database or Docker is down. A dead LM Studio instance, a backed-up scheduler, a filling disk or a degraded knowledge index are all reported in the body but still return 200. So alert on the HTTP status for database and Docker, and parse the body to catch an inference or disk problem.

For an active check that a model still drives tools end to end, use the benchmark harness in §11 — Benchmarking; the Engine no longer exposes a probe endpoint of its own.

The background services

Nine hosted services run alongside the request path, doing the work the Engine manages for you. This is your map of what is running in the background and what "healthy" looks like for each — most of them signal health by a line at startup and then go quiet.

Service What it does Cadence Healthy signal
ModelSchedulerBackgroundService Drains the queue that serializes LLM traffic so the GPU does not thrash Continuous (event-driven); 15 s startup delay Boot line ModelSchedulerBackgroundService starting, then Initial model state refresh completed; no fatal error
LMStudioHealthCheckService Pings each LM Studio instance, updates its health and loaded-model state Every 30 s Boot line started (interval: 30s); thereafter logs only on change (is now healthy / is now unhealthy)
ChromaDbHealthService Probes ChromaDB, auto-reinitializes on recovery Every 2 min Silent when healthy; a marking unavailable warning means an outage, ChromaDB recovered means it healed
ContainerCleanupService Idle → stop → destroy sweep, plus an orphan-resource pass Every 5 min (SystemSettings) Boot line ContainerCleanupService started. Idle timeout: …; stop and destroy actions at Info, per-cycle heartbeat at Debug
WorkspaceCleanupService Archives and deletes tier-expired projects; prunes old archives Every 1 hour Boot line WorkspaceCleanupService started; cleanup actions logged only when projects actually age out
LogIngestionBackgroundService Indexes finished build transcripts into the RAG index Every 2 min; 60 s startup delay Boot line started — scanning every 120s; a scan-cycle failure logs at Warning
KnowledgeSeederBackgroundService Drains the knowledge-seeding queue; provisions curated packs Continuous (event-driven) Boot line started (delay=30s, …) or disabled via config; per job Knowledge seeding completed for project {id}
ToolCatalogValidator Boot assertion that the router's tool catalog matches the prompt-enhancer's Once, at startup [ToolCatalogValidator] OK — {N} tools registered, all documented.
WorkspaceScannerService Boot import of on-disk projects into the database Once, at startup Workspace scan complete: {D} discovered, {I} imported, {S} skipped

Three things about that table change how you read the journal. Two of the nine run once at boot, not on a loop — ToolCatalogValidator and WorkspaceScannerService implement IHostedService directly, while the other seven derive from BackgroundService (§06 — Key Services). Their healthy signal is a single line at startup, not a recurring heartbeat. ToolCatalogValidator can stop the Engine from starting: by default it treats any drift between the routable tool catalog and the prompt-enhancer's as fatal and throws, so a [ToolCatalogValidator] error in the boot log is a hard startup failure, not a warning. And several services are deliberately quiet when healthy — LMStudioHealthCheckService and ChromaDbHealthService log only on state changes, and the two cleanup sweepers emit their per-cycle heartbeat at Debug, below the default Information level. Absence of an error line is the healthy state for these; do not read silence as failure.

Reading the logs

The Engine uses stock .NET logging and writes to standard output — there is no Serilog, no file sink, no log directory to rotate. On the deployed host it runs as the aetherium.service systemd unit (§14 — Deployment), so its output lands in the journal and you read it with journalctl:

# Follow live, starting at the tail
journalctl -u aetherium.service -f -n 0

# The last 200 lines
journalctl -u aetherium.service -n 200 --no-pager

# Everything since a point in time
journalctl -u aetherium.service --since "10 min ago" --no-pager

# Just the errors
journalctl -u aetherium.service -n 500 --no-pager | grep -iE "error|exception"

The default log level is Information, with ASP.NET Core's own categories pinned to Warning so request noise stays out of the journal — §09 — Configuration covers the level keys and the appsettings.Linux.json overlay that sets them. That default is verbose enough to see what the Engine is doing and quiet enough to read.

Two conventions make the logs searchable. Many subsystems prefix their lines with a bracketed tag — [MODEL-ROUTING], [VRAM], [TEMPLATE-DRIFT] — so you can follow one concern by piping through grep:

journalctl -u aetherium.service -f | grep "\[MODEL-ROUTING\]"

And a build run stamps its identifier into the lines it emits (Created BuildRun {RunId} …, BuildRun {RunId} finalized as {Status}). There is no ambient correlation ID tagging every downstream line, so to trace one run end to end you grep the journal for that run's ID or GUID rather than filtering by a scope. When you need the full HTTP request flow for forensics, set AETHERIUM_HTTP_VERBOSE=1 in the service unit's Environment= block and restart — it raises the two ASP.NET hosting categories to Information for that session. Leave it off in steady state; it is deliberately loud.

Data lifecycle and retention

The Engine reclaims some of its own storage and ignores the rest. Knowing which is which is the difference between a host that runs for months and one that fills its disk.

The audit table grows without bound

Every gated tool call is written to PostgreSQL by ToolSecurityAuditService (§07 — MCP Architecture), one row per call in the ToolSecurityAuditEntries table. Nothing prunes it. There is no retention key, no background sweep, and — unlike most project data — these rows are not even removed when their project is deleted. On a busy Engine this table is the single most likely thing to fill your disk.

It is yours to prune. The rows carry a Timestamp column, so a scheduled delete is all it takes:

docker exec -i <postgres-container> psql -U aetherium_user -d aetherium_refactor_db <<'SQL'
DELETE FROM "ToolSecurityAuditEntries" WHERE "Timestamp" < NOW() - INTERVAL '90 days';
SQL

The table name is mixed-case, so PostgreSQL requires the double quotes around the identifier. Schedule the delete from cron at whatever retention your audit needs allow, and run VACUUM (ANALYZE) "ToolSecurityAuditEntries"; after a large first purge to return the freed space to the filesystem.

The tool-audit table has no automatic retention

ToolSecurityAuditService appends to ToolSecurityAuditEntries on every gated tool call and never deletes — not even when a project is removed. Left alone on a continuously running Engine, it grows until it fills the disk. Schedule the pruning delete above; there is no config key that will do it for you.

Container cleanup runs itself

The container lifecycle — idle, then stopped, then destroyed — is swept automatically by ContainerCleanupService (§06 — Key Services), and its three timers live in the SystemSettings table, not in appsettings.json:

Setting Default Effect
Container_IdleTimeoutMinutes 60 Idle minutes before a running container is stopped
Container_DestroyGracePeriodMinutes 30 Minutes stopped before the container is destroyed
Container_CleanupIntervalMinutes 5 How often the sweep runs

Change these from the Settings page; the service re-reads them at the top of every cycle, so no restart is needed. The identically named timers in the Aetherium:Cleanup appsettings block are inert — they look like they should work and do not, a gotcha §09 documents. Projects flagged PersistentContainer opt out of the sweep entirely and live until you remove them.

Workspace tiers age out; archives expire

Project workspaces are reclaimed by tier. WorkspaceCleanupService runs hourly and retires projects whose tier retention has elapsed:

Tier Retention
Production Never (−1)
Staging 30 days
Temp 7 days

The clock is the project's UpdatedAt, not its creation time, so an actively touched project never ages out. When a project does expire, it is archived to a ZIP and its workspace folder is deleted. Projects whose team sets persistent_container: true are skipped by this sweep entirely — their containers belong to ContainerCleanupService's lifecycle above, and the tier-retention pass will not archive or force-remove them. Those archives are themselves pruned: ArchiveRetentionDays (default 30) is the one live key in the Aetherium:Cleanup block, and the same hourly sweep deletes archives older than it. The tier definitions and their resource ceilings are in §09.

Backups

The Engine ships no backup tooling, and nothing it does for you covers disaster recovery. Three stores hold everything that matters, and backing them up is your responsibility.

PostgreSQL holds projects, agents, conversations, runs, and the audit trail — the relational heart of the Engine. Dump it with pg_dump against the database container:

docker exec <postgres-container> pg_dump -U aetherium_user -d aetherium_refactor_db \
  --clean --if-exists --no-owner --no-privileges > aetherium-db-backup.sql

The container name depends on how you brought the stack up — aetherium_postgres is the common name on a Linux host; the Compose default is aetherium-refactor-postgres. Set it once and script the rest.

ChromaDB holds the knowledge-base collections, persisted as SQLite under the volume at /var/lib/aetherium/chroma-data. Because SQLite needs a quiet moment for a consistent copy, stop the container, archive the directory, and start it again:

docker stop <chroma-container>
tar czf chroma-backup.tar.gz -C /var/lib/aetherium chroma-data
docker start <chroma-container>

The workspaces hold the agents' actual output — code, files, scaffolding — under /var/lib/aetherium/workspaces (the production/, staging/, and temp/ tiers, plus knowledge/ and archive/). Archive the tree, or back up selectively by tier if Temp churn is not worth keeping:

tar czf workspaces-backup.tar.gz -C /var/lib/aetherium workspaces

Run the three together on whatever cadence your data warrants. Restoring is the inverse: pipe the SQL dump back through psql (docker exec -i <postgres-container> psql -U aetherium_user -d aetherium_refactor_db < aetherium-db-backup.sql), un-tar the ChromaDB volume with its container stopped, and un-tar the workspace tree.

Recovery and rollback

Every deploy stamps the live Git SHA into /opt/aetherium/deployed-commit.txt (§14 — Deployment). That file is your anchor for knowing what is running and for rolling back:

cat /opt/aetherium/deployed-commit.txt

Capture that SHA before any risky deploy. To roll back, check the known-good commit out and redeploy what is on disk rather than pulling forward:

git checkout <good-sha>
./scripts/deploy.sh --no-pull

The deploy republishes, re-stamps the marker, restarts the service, and health-checks it — the same path a forward deploy takes. One hazard makes a rollback more than a binary swap: the Engine applies pending database migrations automatically on startup (§14), so by the time you want to roll back, the schema may already be ahead of the older binary you are rolling back to, and there is no down-migration on startup.

Back up the database before any deploy that carries a migration

Auto-migration on startup means the schema can move forward before you decide to roll back, and the older binary may not run against the newer schema. Take a pg_dump immediately before deploying any change with a new migration, and treat a cross-migration rollback as restore-the-database-too, not just a checkout-and-redeploy.

Operational security

Deployment (§14 — Deployment) established a security boundary; operations is keeping it intact.

The orchestrator API stays behind its boundary. The build endpoints — POST /api/orchestrator/run and /api/orchestrator/run-from-template — have no built-in authentication; they run agents that execute shells and write files, and they trust whatever reaches them (§13 — Integrations). The reverse proxy or private-network boundary you set up at deploy time is the only thing protecting that surface. Do not let it erode — every change to the network in front of the Engine is a change to its security posture.

Rotate the secrets. Two live in the deployment and should turn over on a schedule: the PostgreSQL password in appsettings.Local.json, and the HMAC secret each webhook target signs with (Aetherium:Bridge:Webhooks:Targets). Both follow the repo's REPLACE_ME_VIA_LOCAL_OVERRIDE convention. Rotating the webhook secret means updating the Engine and the receiving listener together, since the listener verifies the signature with the same shared secret (§13).

Treat webhook delivery as best-effort across restarts. The webhook dead-letter queue — where failed deliveries wait for retry — lives in memory and is cleared on restart (§13). An event still pending retry when the Engine restarts is lost. For anything a downstream system must not miss, reconcile against the runs API (GET /api/orchestrator/runs/{projectId}/{runGuid}) rather than relying on the webhook stream alone.

Production hardening beyond this — OS patching, proxy configuration, firewall rules, TLS certificates — is yours to own; the Engine assumes the host around it is maintained (§14).

Where to go next