11 — Benchmarking

How the Engine's behavior with a given model is measured — and how to test whether a model works in your own deployment.

The Aetherium Engine is only as reliable as the models behind its agents, and local models vary enormously in the capability the Engine leans on hardest: calling the right tool, with the right arguments, every time. A model that writes flawless prose can still emit malformed tool calls, route to the wrong tool, or drop a required argument — and the orchestration loop stalls the moment a tool call is unparseable. Worse, a single run tells you little: a model can ace one pass and loop on the next, so the only honest way to separate a capable model from a lucky one is repetition. Benchmarking is how you measure both — correctness and consistency — before you trust a model with a team.

More than one thing in the project wears the word "benchmark," so it is worth naming them. The Engine's repository ships a small Aetherium.Benchmarks project, but that is BenchmarkDotNet micro-benchmarks of .NET internals — JSON serialization, slug generation — unrelated to models and outside the solution build, as §10 — Development Workflow noted. The Engine also exposes a lightweight tool-call reliability probe at its /api/benchmark endpoint for a quick in-process check. The real model-and-team benchmark suite, and the subject of this section, lives in a separate benchmarks repository: a Python harness that drives a running Engine over its REST API and scores what comes back. The two repositories are joined by one artifact — the benchmark model registry (model-registry.json): the harness generates it, and the Engine reads it at runtime to route each agent role to its best-scoring model. You benchmark in one repository; the Engine routes on the result in the other. This section covers the producing side; §09 — Configuration covers the consuming side.

The harness

The harness is a Python toolset under the benchmarks repository's runner/ directory. It does not test a model in the abstract — it loads the model into LM Studio, points it at the Engine's live tool schemas, and runs it through a graded progression of tasks that mirror what the Engine actually asks of agents. Each tier is its own runner module (tier0_runner.py through tier5_runner.py); the tool-calling tier is the main benchmark_runner.py.

Two JSON files anchor everything, and keeping them straight is the key to understanding the harness:

• runner/config.json is the input. It lists the candidate models the harness will load and test, their quantization and load parameters, the context lengths to sweep, the system-prompt variants, and the repetition count. This is the roster you edit to benchmark a new model.
• model-registry.json is the output. It is generated from a run by runner/registry_generator.py, never hand-maintained as a roster. It holds each benchmarked model's aggregate scores, its per-task scores, and a default_routing map naming the best model for each task type. This is the file the Engine consumes.

Data flows one way: config in, registry out.

A run is invoked per tier. The tool-calling tier looks like this:

Windows (PowerShell)Linux (bash)

cd Aetherium-benchmarks
python runner\benchmark_runner.py --new --model "your-model-id"

cd ~/aetherium-benchmarks
python3 runner/benchmark_runner.py --new --model "your-model-id"

--new starts a fresh run; --resume continues an interrupted one (the two are mutually exclusive). --model narrows the run to a single model, --machine to one host, and --dry-run validates the setup without calling the model. Selecting which tier to run means choosing which runner to invoke; selecting what within a tier uses tier-specific flags — --config and --scenario for the integration tiers, --pipeline for the pipeline tier. Repetitions are not a flag — they live in config.json (the project runs three by default), because reliability is measured across repeats, not requested per invocation.

Results land under results/ (one subtree per tier, keyed by model or config and date), with run state and a human-readable summary.md under a sibling runs/{run-id}/ directory. A separate baselines.json records known-good numbers from prior runs so a new run can be checked for regressions — it is a reference, not an input. None of this is the model registry: results capture what happened; the registry captures the routing verdict.

flowchart LR
    CFG[config.json<br/>candidate roster] --> RUN[tier runner]
    ENG[running Engine] -->|live tool schemas| RUN
    RUN -->|loads model| LM[LM Studio]
    RUN -->|drives builds| ENG
    RUN --> RES[results/ + runs/]
    RES --> GEN[registry_generator]
    GEN --> REG[model-registry.json]
    REG -->|consumed by model routing| ENG

The benchmark lifecycle: the candidate roster feeds a tier runner, which loads the model and exercises it against the live Engine; scored results are compiled into the model registry the Engine then routes on.

The benchmark tiers

The suite is six tiers, numbered 0 through 5. They climb from "can this model form a single tool call" to "can four teams hand a project down a pipeline without losing the thread." Each tier is a cheaper gate on the one above it: you do not spend hours on an end-to-end build for a model that cannot pass a five-minute smoke test.

Tier	What it measures	What a pass looks like
0 — Quick-Check	Whether the model can operate the machinery at all	All three smoke tests pass — a hard gate, not a score
1 — Tool calling	Single-shot tool selection and argument correctness	A high mean score with low variance across repeats
2 — Code generation	Whether generated code is correct and runnable	Code passes hidden tests, plus a judge quality rating
3 — End-to-end build	Whether a team can build a working app via the live orchestrator	A high share of behavioral checks pass against the running build
4 — Quality	Documentation and finished-code quality, including faithfulness to source	Structural checks plus a calibrated judge score
5 — Pipelines	Whether facts survive multi-team handoffs	Planted facts propagate end-to-end above a threshold

A few tiers carry detail worth knowing:

• Tier 0 is a compatibility gate, not a capability score. Its three tests — one tool call, one context-transforming tool call, one minimal end-to-end build — answer a single question: does the model operate the harness at all? Failing the first means the model cannot produce a usable tool call and is incompatible with the Engine, full stop; there is no point scoring it further. It runs in about five minutes and exists to spare you the expensive tiers.
• Tier 1 grades on a 0–5 ladder rather than pass/fail: 5 for the right tool with every required argument, 4 for an acceptable alternative tool, 3 for the right tool with arguments missing, 2 for the wrong tool, 1 for an unparseable call, 0 for no call at all. The required arguments come from the Engine's live tool schemas, so the test always matches the real contract.
• Tier 2 runs the model's code against tests it never saw. The model writes code through tool calls; the harness stages a hidden test file, runs it in a Docker sandbox, and scores execution (did the tests pass) separately from a language-model judge's quality rating. "Looks right" and "passes the hidden tests" are scored as two different things.
• Tier 3 is pure behavior, no judge. It drives the real orchestrator to build an app, then probes the running result — health endpoints, CRUD operations, files on disk — and reports the share of checks that passed. Failures are classified as infrastructure, Docker, or model-quality, so a flaky container is not blamed on the model.
• Tier 4 measures quality of documentation and finished code, and weights faithfulness heavily: checks that catch a doc claiming a flag or behavior the source does not have count double. Hallucinated content is treated as worse than missing content.
• Tier 5 measures pipeline coherence. It plants distinctive "canary" facts — a specific port, a flag name — in the source a first team sees, then checks whether those facts survive the handoffs to downstream teams, carried through a shared knowledge base. The headline metric is the canary survival rate; a pipeline sets a minimum (for example, 75%). Each step can also declare tool_defaults that force default arguments, such as the knowledge-base collection a handoff must write to.

Two of the scored tiers lean on a language model as a judge, and the harness does not trust it blind. Before a judge's scores count, it must rank a fixed ladder of known good-to-bad samples in the correct order. If that calibration fails, the judge's scores are suppressed and the tier falls back to structural checks alone — a guard against a miscalibrated judge inventing quality.

Reading a benchmark result

A benchmark result is a shape, not a single number, and the shape differs by tier. Tier 1 reports a mean score per model alongside its standard deviation and a per-tool breakdown — and it is the standard deviation that matters most. A tool whose scores swing run to run (the harness flags any tool with a standard deviation above 0.5 as inconsistent) is one your agents cannot depend on, even if its average looks strong. This is the single-pass-versus-reliability distinction made concrete: the average tells you the model's ceiling, the variance tells you whether it reaches that ceiling every time. Run at least three repetitions — one pass cannot show you variance.

The integration tiers report a pass count over a total, a pass rate, and — for code and docs — a composite score out of five, split into an execution (or structural) part and a judge-quality part. A result also carries the individual checks, each with an ID, a pass/fail, and a message, so a failure points at the exact thing that broke rather than a bare score. The pipeline tier reports a per-step breakdown and a canary survival map: you can see which fact died at which handoff.

Looks like success, scores zero

Some inference adapters report a finish_reason of stop even when the model emitted a tool call, and some dump a JSON tool call into the message text instead of the structured tool-call field. The harness detects tool calls by inspecting the structured field, not the finish reason, and treats a JSON blob in the content as a failure — because that is exactly what the Engine's orchestration loop would choke on. If a model "looks like it answered" but scores zero, this is usually why.

Adding a benchmark case

Every tier reads its cases from disk, so extending the suite is a matter of writing files in the right format. The format escalates with the tier, but the most representative case to add — and the one closest to a self-contained "benchmark case" — is a Tier 2 code-generation task. It is a single JSON object with three parts: the prompt and tool expectations, an execution block, and a rubric.

{
  "id": "cg-demo",
  "prompt": "Write reverse_str.py exposing reverse(s) that returns s reversed.",
  "primary_tool": "write_file",
  "execution": {
    "language": "python",
    "test_content": "from reverse_str import reverse\n\ndef test_basic():\n    assert reverse('abc') == 'cba'\n",
    "test_command": "python -m pytest -q",
    "timeout_seconds": 30
  },
  "rubric": {
    "correctness": { "weight": 3, "checks": ["all_tests_pass"] },
    "style":       { "weight": 1, "checks": ["has_type_hints"] }
  }
}

Illustrative only — a trivial case showing the schema, not a real benchmark fixture.

The execution block is the validator: the harness applies the model's tool calls in a Docker sandbox, stages your test_content (which the model never sees), runs test_command, and turns the pass ratio into the execution score. The rubric drives the language-model judge for the quality score. To register the case, you append it to the tier's tests.json — the runner discovers the whole array automatically, and you can target your new case alone with --test cg-demo. (You can also author cases as Markdown files with YAML frontmatter and compile them to JSON with the harness's converter, if you prefer prose to raw JSON.)

The other tiers follow the same idea with different mechanics:

• Tier 0 cases are one JSON file each, carrying a pass_criteria block of structural assertions — expected tool, path pattern, required substrings. They are wired into the runner explicitly rather than auto-discovered, so adding one means adding a load line.
• Tiers 3 and 4 define a scenario in the tier's config file, but their validators are Python: each scenario has a validate_* function of behavioral checks, dispatched by scenario ID. Adding a scenario means writing that function and wiring it in — a code change, not just a fixture.
• Tier 5 is the cleanest extension point: a pipeline is a single JSON file in tier5/pipelines/ declaring its steps, its planted canaries, and its handoff and output checks. The validator is generic and reads those blocks, so a new pipeline needs no Python at all.

There is no JSON Schema file to validate against — the contract lives in the runner and validator code, which is the place to look when a field is ambiguous.

Evaluating a new model

The reason to run any of this is usually a concrete question: you downloaded a model in LM Studio — does it work in your Engine? The workflow is a gated progression, cheapest gate first.

1. Pre-load the model in LM Studio. The harness reaches LM Studio at http://localhost:1234 over its OpenAI-compatible and native APIs, and manages VRAM itself — it unloads any resident model and keeps one model loaded at a time, because everything runs against a single inference host. Set the model to maximum GPU offload, and confirm its exact ID with a quick GET /v1/models; the ID you pass the runner must match that string.
2. Register it in the input config. Add an entry to runner/config.json with the model's ID, quantization, and load parameters. The model registry is generated later, from results — you do not hand-write it.
3. Run the gate, then climb. Start with Tier 0; if it fails the first test, the model is incompatible and you are done. If it passes, run Tier 1, then promote to the heavier tiers only if the lighter ones earn it. A preflight script checks that the Engine, LM Studio, and the test corpus are all reachable before a run starts.

Windows (PowerShell)Linux (bash)

python runner\tier0_runner.py --model "your-model-id"

python3 runner/tier0_runner.py --model "your-model-id"

1. Read the score against the role. There is no single "good model" verdict — there is "good enough for this role." A supervisor lives or dies on tool-routing reliability (Tier 1 variance), a developer on code that passes Tiers 2 and 3, a doc writer on Tier 4. The project's own promotion rule is concrete: a model earns a tool-calling role when its Tier 1 mean clears roughly 4.5 with a standard deviation under 0.5 (the same 0.5 the harness uses to flag an inconsistent tool), and a code-generation role when its Tier 2 composite clears roughly 3.5 out of 5. Treat those as bars to clear on your own hardware, not guarantees — and before changing a production default, compare the new model against your current baseline, because a model that aces Tier 1 can still fail Tier 3.

In Claude Code, the aetherium-model-benchmark skill automates this path. It lives in .claude/skills/ in the repository you cloned — the same place as the update-memory-bank skill from §10 — Development Workflow — and it encodes the whole methodology: the preflight checklist, the per-tier commands, the add-a-model sequence, the judge-calibration rules, and the regression-comparison legend. It drives the runners and reads back the scorecards, so "benchmark this model and tell me which role it fits" becomes one request rather than an afternoon of manual runs.

Where a chosen model actually gets wired into the Engine is a configuration question, not a benchmarking one. §09 — Configuration covers the model-resolution chain, including the registry-routing tier that consumes the file this harness produces and the operator settings (AI_ModelId and the rest) that can override it. The service that then serializes that model's traffic so concurrent agents do not thrash the GPU is the ModelSchedulerService in §06 — Key Services.

Where to go next

• §06 — Key Services — the model scheduler and LLM router the benchmarks drive, and the VRAM planner that keeps a run from evicting its own models.
• §09 — Configuration — model resolution, including the benchmark-registry routing tier and the operator settings that wire a chosen model into the Engine.
• §12 — Extending the Engine — adding the tools, roles, and templates a new benchmark case would then cover.