14 — Deployment¶

Getting the Engine off your dev box and onto a Linux host your team or your pipeline can point at.

This section is the transition from dotnet run on a dev box (§02 — Getting Started) to a deployed Engine running as a managed service on a Linux host. The build mechanics underneath it are §10 — Development Workflow; the keys you set are §09 — Configuration; ongoing care of a running host is §15 — Operations. This section owns the how to deploy: the publish, the dependencies, the service, and the boundary you put in front of it.

The canonical deployment of The Aetherium Engine is a single Linux host. One machine runs the Engine as a systemd-managed process, PostgreSQL and ChromaDB as Docker containers, and an inference server — LM Studio or any OpenAI-compatible endpoint — either co-located or reachable across your LAN. An optional reverse proxy sits in front. Everything the Engine needs is on one box, and that is the shape that has been built, run, and debugged.

Deployment is genuinely Linux. Windows is the development and planning environment (§05 — Solution Layout, §10); there is no production Windows deployment path, and the commands below are bash and systemd by design — you will not find PowerShell equivalents, because in production there is nothing to run them against. The components can be separated across hosts (covered at the end), but the documented, working path is single-host. Treat multi-host as a configuration you can reach for, not a topology that has been hardened.

flowchart TB
    CLIENT["Your orchestrator / team<br/>(trusted network)"]
    PROXY["Reverse proxy<br/>TLS · identity · generous timeouts"]

    subgraph HOST["Single Linux host"]
        ENG["The Engine<br/>systemd: aetherium.service<br/>:5257"]
        subgraph DOCKER["Docker"]
            PG[("PostgreSQL<br/>:5432")]
            CH[("ChromaDB<br/>:8000 · ext4 volume")]
        end
        LM["LM Studio<br/>:1234 (local or LAN)"]
    end

    CLIENT -->|HTTPS| PROXY
    PROXY -->|"http :5257"| ENG
    ENG --> PG
    ENG --> CH
    ENG -->|"/v1 inference"| LM

    classDef engine fill:#B87333,stroke:#2D2D2D,stroke-width:2px,color:#FFFFFF
    class ENG engine

The single-host topology. The Engine and its datastores live on one machine; inference is local or LAN-reachable. Everything inbound crosses the reverse proxy, which is where identity and TLS are enforced — the Engine itself trusts whatever reaches it.

Prerequisites¶

The host needs five things before the first deploy:

Need	Why
.NET 10 SDK	The deploy script publishes on the host — it builds, so it needs the SDK, not just the runtime (§02)
Docker + Compose	Runs PostgreSQL and ChromaDB
An OpenAI-compatible inference server	LM Studio or any `/v1` server, local or LAN-reachable (§13 — Integrations)
A POSIX-locking filesystem for the ChromaDB volume	ChromaDB's SQLite backend needs advisory file locks
A dedicated service user (for example `aetherium`)	Owns `/opt/aetherium` and runs the service under its own identity

ChromaDB's data volume must support POSIX file locking

Put the ChromaDB volume on ext4, btrfs, or xfs — never ntfs3. ChromaDB persists each collection as SQLite, and SQLite relies on POSIX advisory locks that the Linux NTFS driver does not honor; on an NTFS mount every POST /collections returns 500 ("unable to open database file", SQLite code 14). The compose file already defaults the volume to /var/lib/aetherium/chroma-data and raises the container's open-file limit to 65536 — that ulimit is there because each collection holds several file descriptors and the Docker default of 1024 hits the same error under load. Keep both. This advisory-lock issue is distinct from the assembly-memory-mapping quirk that affects building the Engine on NTFS (§10) — same filesystem, two different problems.

Publishing the Engine¶

Publish the server .csproj directly — never the .slnx. This is the canonical gotcha from §10: dotnet publish against the solution file builds every project but never runs the publish target on the server, so you get no runnable server output. Always name the project:

dotnet publish Aetherium_Refactor/Aetherium_Refactor/Aetherium_Refactor.csproj \
    -c Release -o /tmp/aetherium-publish

The publish is framework-dependent — it does not bundle the .NET runtime — which is why the host needs the .NET 10 runtime present (the SDK already provides it). If you deploy from a freshly cloned tree, apply the one-time build-directory fix from §10 before the first publish: a fresh Linux clone hits a glob-expansion error until bin/ and obj/ exist, and the publish runs from the working tree.

In practice you rarely run dotnet publish by hand — the deploy script wraps it. The standalone command is here so you know exactly what the script does.

Provisioning PostgreSQL and ChromaDB¶

The datastores run from the same docker-compose.yml you use in development — there is no separate production compose file. From the repository root:

cd ~/aetherium-core
docker compose up -d
docker ps

You get two containers: aetherium-refactor-postgres (PostgreSQL 15 on port 5432) and aetherium-chromadb (ChromaDB on port 8000, published on 127.0.0.1 only and pinned to chromadb/chroma:1.5.2 by digest, so a backup and its restore always run on the same Chroma — an upgrade is a deliberate edit with its own restore drill). PostgreSQL holds projects, agents, conversations, and logs; ChromaDB holds the per-project knowledge collections. The PostgreSQL data volume defaults to ./postgres-data and the ChromaDB volume to /var/lib/aetherium/chroma-data; override either with the AETHERIUM_POSTGRES_DATA and AETHERIUM_CHROMA_DATA environment variables, keeping the ChromaDB target on a POSIX-locking filesystem.

The compose file ships with the development defaults aetherium_user / aetherium_password and publishes PostgreSQL on 5432. Change the password for any host where the port is reachable beyond loopback, and set the matching connection string in the next step. The SearXNG search backend (§13) is a separate, optional compose file and is not part of this stack. Bringing up the dev datastores is covered in §02; the connection keys are in §09.

Configuring for production¶

Two files carry production configuration, and they layer.

The committed appsettings.Linux.json is the per-environment overlay. It loads only when ASPNETCORE_ENVIRONMENT is set literally to Linux — not merely because the host runs Linux, a gotcha worth re-reading in §09. When active it points storage at /var/lib/aetherium/workspaces, sets the ChromaDB endpoint to http://localhost:8000, and runs at Information log level. The service unit below sets that environment variable, which is what makes the overlay take effect.

The gitignored appsettings.Local.json is the highest-precedence file layer and the home for anything host-specific or secret — above all the real PostgreSQL connection string. Secrets use the repo convention REPLACE_ME_VIA_LOCAL_OVERRIDE; never commit a real value.

{
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Port=5432;Database=aetherium_refactor_db;Username=aetherium_user;Password=REPLACE_ME_VIA_LOCAL_OVERRIDE"
  }
}

Storage paths beyond the Linux overlay use the ${ENV_VAR} interpolation pattern — ${AETHERIUM_WORKSPACE_ROOT} and its siblings expand against the environment at startup, falling back to /var/aetherium/... on Linux when unset (§09). That is the mechanism for pointing one committed config at host-specific drives. Everything else that tunes runtime behavior — container idle and cleanup timers, model selection, feature flags — lives in the SystemSettings table, changeable from the Settings page or the API with no restart; the services that read those rows are described in §06 — Key Services.

Installing the systemd service¶

The Engine runs as a systemd service. The repository does not ship a unit file — it is an install convention — so you create one. The unit below reflects the fields the running deployment depends on; the install-specific values (the service user, the dotnet path) are yours to set.

# /etc/systemd/system/aetherium.service
[Unit]
Description=The Aetherium Engine
After=network.target docker.service
Wants=docker.service

[Service]
Type=simple
User=aetherium
Group=aetherium
WorkingDirectory=/opt/aetherium
Environment=ASPNETCORE_ENVIRONMENT=Linux
Environment=DOTNET_ROOT=/usr/lib/dotnet
ExecStart=/usr/bin/dotnet /opt/aetherium/Aetherium_Refactor.dll
Restart=on-failure
RestartSec=5
SyslogIdentifier=aetherium

[Install]
WantedBy=multi-user.target

Three directives are load-bearing:

WorkingDirectory=/opt/aetherium — ASP.NET Core reads appsettings*.json from the working directory, not from the binary's location. Omit this and the Engine launches but loads none of your config.
DOTNET_ROOT and the ExecStart path must both point at your actual .NET install. The example assumes a system-wide install (/usr/bin/dotnet, /usr/lib/dotnet); if you installed the SDK under the service user's home directory instead, use /home/aetherium/.dotnet/dotnet for both. A mismatch is the classic crash-loop — the service starts, can't find the runtime, and restarts forever.
ASPNETCORE_ENVIRONMENT=Linux — this is what loads appsettings.Linux.json.

The install directory is /opt/aetherium, owned by the service user so the deploy script can sync into it without sudo. Create and claim it, then enable the service:

sudo mkdir -p /opt/aetherium
sudo chown -R aetherium:aetherium /opt/aetherium
sudo systemctl daemon-reload
sudo systemctl enable --now aetherium.service
sudo systemctl status aetherium.service

Deploying with the deploy script¶

scripts/deploy.sh is the repeatable path from a Git commit to a running service. It is idempotent, refuses to deploy a dirty tree, and verifies health before reporting success. Run it from the repository root:

./scripts/deploy.sh --dry-run   # report what would happen, change nothing
./scripts/deploy.sh             # do it

A normal run walks these steps: locate dotnet; refuse to proceed if the working tree has staged or modified changes (untracked files are ignored) unless --force; git pull --ff-only; build the React client (Aetherium.Web) with bun into the server's wwwroot/app — before the publish, because assets that appear after it are not served — where bun is looked for on PATH, in $BUN_INSTALL/bin and in ~/.bun/bin, and the script exits with status 7, nothing changed, when none has it; publish the server .csproj in Release to a temporary directory; check, before anything in /opt/aetherium changes, that the config about to be deployed can pin every GPU unit to a card (scripts/render-gpu-env.py --validate: the slot whose Unit is llama-server.service must name a Card that Aetherium:Gpus:Cards lists and sysfs reports, and the binary its build names must exist), exiting with status 6 and nothing changed when it cannot — and the same check on each machine a card's Host names, over SSH, against only the config sections the script reads (render-gpu-env.py --export, no connection strings), where a machine that does not answer is skipped with a warning and is neither checked nor synced this deploy; preserve any appsettings.Local.json overrides from the live install across the sync, and copy each one to ~/.config/aetherium/config-backups/ (or $AETHERIUM_CONFIG_BACKUP_DIR), named with the UTC time and the previously deployed commit, mode 0600, keeping the newest 20 of each file; rsync --delete the publish output into /opt/aetherium; restore the preserved overrides; run scripts/gpu-node-units.sh for the cards with no Host, which renders each GPU unit's card-pin environment file into ~/.config/aetherium/gpu/ (every GPU unit requires its file and will not start without one; the file also sets AETHERIUM_LISTEN, the address the unit's server binds); install the leased GPU service units from the publish output's services/ directory into the user's systemd (never enabled — the inference arbiter starts and stops them; a running one whose pin changed is reported, not restarted); install the two llama-server router units and their preset files from the publish output's llama/ directory — llama-server.service (Primary, models.ini, port 8090), enabled and kept running, and llama-server-worker.service (models-worker.ini, port 8091), installed but left for you to enable — restarting a router only when its unit, presets or pin changed or it is not running (a restart drops every model it holds; the Engine re-warms the slot's standard set within about a minute); for each card Host that answered, sync llama/ and services/ into that machine's /opt/aetherium, write the exported config to /opt/aetherium/gpu-node/appsettings.json there (its own appsettings files are left alone) and run the same script there with --host; stamp the deployed commit SHA into /opt/aetherium/deployed-commit.txt; sudo systemctl restart aetherium.service; then poll http://localhost:5257/api/health for up to 60 seconds. If the service does not come back healthy, the script exits non-zero and prints the journalctl command to diagnose it.

Two behaviors are worth internalizing. The override preservation is why your secrets survive a redeploy: the script copies the *.Local.json files out before the rsync --delete and restores them after, and the same files are excluded from the sync — which is also why a Skills/ folder placed inside /opt/aetherium would not survive, and belongs at an external path instead. The commit marker records exactly which commit is live; it is the anchor for knowing what you are running and for the rollback procedure in §15. Useful flags: --no-pull (deploy what is checked out), --no-restart (publish and sync without bouncing the service), --no-spa (skip the client build and keep the previously built client — needed once when adding an endpoint, because the client's TypeScript types are generated from the running server's OpenAPI document), and --force (deploy over a dirty tree).

The install layout¶

After a deploy, /opt/aetherium holds the published Engine plus the two files the script manages:

Path under `/opt/aetherium`	What
`Aetherium_Refactor.dll` + dependency DLLs	The server binary and its references — what `ExecStart` runs
`wwwroot/`	Static assets, including `app/` — the compiled React client
`appsettings.json`, `appsettings.Linux.json`	Committed config — overwritten on every deploy
`appsettings.Local.json`	Your gitignored overrides — preserved across deploys
`deployed-commit.txt`	The Git SHA currently deployed

The wwwroot/ folder is the reason you publish rather than build: a plain dotnet build output omits it, and the Engine would serve the API but return 404 for the browser client. The Workflow Templates need no entry here — they ship as embedded resources inside the binary, not as loose files (§08 — Templates). The one thing deliberately not in this directory is the Skills/ folder, for the reason above.

First start¶

When the service starts, pending database migrations apply automatically through Database.MigrateAsync() — there is no manual dotnet ef database update step in production (§10). Bring the datastores up first: the compose stack has no PostgreSQL health gate, so if the database is not yet accepting connections when the Engine starts, migration logs an error and the Engine runs degraded. Start Docker, confirm PostgreSQL is ready, then start the service.

Confirm the deployment is live:

curl http://localhost:5257/api/health
sudo journalctl -u aetherium.service -n 50 --no-pager

A healthy response means the database is reachable, migrations have applied, and the API is serving — the same signal §02 uses on a dev box. Scan the startup log for the two things that most often go wrong on a fresh deploy — an unrotated secret placeholder or an unhandled exception:

sudo journalctl -u aetherium.service -n 100 --no-pager | grep -iE "REPLACE_ME|error|exception"

Then confirm the Engine can reach inference by listing what the server is serving:

curl -s http://localhost:1234/v1/models | jq '.data[].id'

If the health check fails or the service crash-loops, the deploy-time failure modes are in §16 — Troubleshooting; the DOTNET_ROOT mismatch and the working-directory config-load issue above account for most first-start failures.

Securing the deployment¶

The orchestrator API has no built-in authentication. The endpoints that drive a build — POST /api/orchestrator/run and /api/orchestrator/run-from-template — trust whatever calls them; the Engine assumes it sits inside a trusted network boundary, and at deploy time that boundary is yours to build (§13).

So put the Engine behind a layer that enforces identity. A reverse proxy with mTLS, an API gateway, or a bearer-token check at the edge all work; so does keeping the Engine on a private network with no external route at all. The proxy in the topology diagram is where TLS terminates and identity is checked — the Engine listens on plain HTTP on 5257 behind it.

The Engine itself listens on every interface, because agent containers reach it through their per-project Docker bridges (host.docker.internal). To keep everything else off the port, the repo ships an optional nftables guard: scripts/net/aetherium-guard.nft, loaded by scripts/net/aetherium-guard.service, accepts :5257 on lo, tailscale0, docker0 and the br-* bridges and drops it on every other interface. It is its own table, so it changes nothing else, and stopping the unit deletes it. Installing it needs root:

sudo nft -c -f scripts/net/aetherium-guard.nft      # syntax check, changes nothing
sudo install -D -m 644 scripts/net/aetherium-guard.nft /etc/aetherium/aetherium-guard.nft
sudo install -m 644 scripts/net/aetherium-guard.service /etc/systemd/system/aetherium-guard.service
sudo systemctl daemon-reload && sudo systemctl enable --now aetherium-guard.service

When the cards are in another machine (a card Host), two more pieces belong on that machine. The Engine's SSH key (Aetherium:Gpus:RemoteSshKey) goes into its ~/.ssh/authorized_keys pinned to the wrapper — command="/bin/bash /opt/aetherium/services/aetherium-gpu-node",restrict ssh-ed25519 AAAA… — so it can run nothing but is-active, start, stop and write-env on the GPU service units the deploy published there, and cards, which prints that machine's GPUs from sysfs for card telemetry. And because the routers and GPU services there bind the card's ListenAddress instead of loopback, scripts/net/aetherium-gpu-node-guard.nft (loaded by aetherium-gpu-node-guard.service, installed the same way as the guard above) accepts their ports only from loopback and the Engine's machine; set the Engine's address in the file first.

Separately, the knowledge routes that drop collections or delete the library (POST rag/rebuild, DELETE rag/index, DELETE global/folders/{f}, DELETE packs/{name}, DELETE collections/{c}) answer 403 to any caller that is not on the Engine's own machine (API · Knowledge).

If you front it with a reverse proxy, set generous read timeouts. Both run endpoints are synchronous: they hold the connection open through an entire multi-agent build, which can run for minutes. A proxy with a default 30- or 60-second read timeout will sever long builds mid-flight and return a gateway error to the caller while the Engine keeps working. Size the timeout to your longest expected build (§13).

Never expose the orchestrator API directly to the public internet

An unauthenticated build endpoint reachable from the open internet is an arbitrary-code-execution surface — every agent can run shells and write files inside its container. Keep the API on a private network or strictly behind an authenticating proxy. Outbound authenticity (signed webhooks) is handled by the Engine; inbound authenticity is entirely your deployment's responsibility. Ongoing operational security is §15 — Operations.

Separating the components¶

Single-host is the documented path, but the pieces are not welded together. PostgreSQL, ChromaDB, and the inference server can each run on their own host: point ConnectionStrings:DefaultConnection, ChromaDb:Endpoint, and the inference URL (§09) at the remote addresses, all reachable over your private network — the egress gate already permits RFC 1918 ranges (§13). You can likewise publish in CI and copy only the artifacts to the host, in which case the host needs the ASP.NET Core 10 runtime rather than the full SDK. What is not yet a documented story is a hardened multi-host or high-availability topology — there is no Engine clustering or load-balanced front end. Separation is supported configuration; treat it as such, not as a turnkey scale-out.

Where to go next¶

§15 — Operations — health, logs, backups, rollback, and keeping a deployed host running.
§09 — Configuration — the production config keys, the precedence order, and the SystemSettings runtime layer.
§16 — Troubleshooting — deploy-time failure modes and how to read them.
§13 — Integrations — driving the deployed Engine over the API, and the boundary it assumes.