Evaluations

Every number Dispach states in public has a row here naming the committed file it comes from. Negative results are listed beside the positive ones, and a claim without evidence is marked as such rather than left out. CI fails when a verified claim's file goes missing.

30 claims: 23 verified, 1 cited from external research, 5 waiting on a run, 1 withdrawn.

Harness head-to-head

  • Dispach native scores 125/125 on five of the eleven models compared with other harnesses, and 99–100% on eight of fourteen models

    verified

    evals/harness/summary.json, evals/harness/runs/sweep-4f3e00a/ (2026-10-23)

    One benchmark of our own design

  • Matches or beats Hermes 0.19.0 on 8 of 11 models, and costs the least of the three harnesses on 10 of 11 (level with Hermes on GPT-6.1 Sol)

    verified

    evals/harness/summary.json

    Rival arms measured 2026-10-22/23, Dispach on 2026-10-23; the 95% intervals of every Dispach–Hermes pair overlap, gpt-oss-20b only just (70–84% against 82–93%)

  • OpenClaw 2026.10.5 scores lowest on every model, 11–60 runs behind

    verified

    evals/harness/summary.json

    Each harness on its own documented defaults

  • Hermes leads on the small open models: qwen3.5-9b 112 vs 104, gpt-oss-20b 111 vs 97

    verified

    evals/harness/summary.json

    The qwen3.5-9b intervals overlap widely; the gpt-oss-20b ones only just

Tool dialect (NLT vs native)

  • NLT ahead of native: qwen3.5:9b 94.6% vs 91.9%, deepseek-chat 97.3% vs 86.5%, deepseek-reasoner 91.9% vs 89.2%

    verified

    evals/tools/results.json, evals/tools/qwen-3pass/results.json (2026-08-14)

    No tool is executed. The qwen margin is one fixture. qwen NLT has a 2.7% critical rate against native's 0.0%

  • Native ties or beats NLT on every model on multi-turn tool work; NLT falls to 50–68% on Grok 4.7, Kimi K3, Gemma 4 26B-A4B and GPT-6 Luna

    verified

    evals/harness/summary.json, evals/harness/runs/sweep-4f3e00a/results.json (2026-10-23)

    One benchmark; qwen3.5-9b under NLT paused at 52 of 125 runs

  • NLT uses 21–23% fewer prompt tokens

    verified

    evals/tools/results.json

    Prompt tokens only, not total cost

  • +14.9pp accuracy, 93% fewer critical errors, −25% tokens; +24 to +43pp on small models

    cite

    Somma, Plante & Premji, 2026 replication of Johnson et al. 2025, arXiv 2607.03953, 14 models, 8,560 trials

    Preprint. Frontier models (GPT-5, Gemini 2.5 Pro) show smaller or reversed gains. Our own eval cannot test the critical-error claim. Not reproduced on multi-turn work: in evals/harness/runs/sweep-552660a/, native tied or beat NLT on 13 of 14 models, NLT 37–62% on five

  • Decision 4.1's local deltas

    verified

    evals/tools/results.json

    4.1 now quotes the committed +10.8 / +2.7pp and says the earlier +13.5 / +5.4pp run was overwritten

Phase-scoped tools

  • Constraining tools per phase took local models from 2/10 to 10/10

    remove

    None found in the repo or in a search

    Decision 4.5 now names it as unsourced motivation beside the negative local result. Never quoted publicly

  • On deepseek-v4-pro, phase scoping cost 8.3–12.5pp

    verified (negative)

    evals/phases/results.json

    Frontier model only; the small-model arm has never been run. Run-to-run noise was 4.2pp

Boot and runtime

  • Boot to runtime.ready under 1000 ms, CI fails at 1200 ms

    verified

    scripts/bench-boot.ts:18, .github/workflows/ci.yml

    Boot, not response latency

  • Boot to ready ~87–91 ms (0.2.0; ~53 ms at 0.1.3)

    verified

    evals/boot/results.json (bench:boot --write)

    Boot, not response latency. One machine

  • Container image 867 MB

    verified

    docker image inspect, 2026-10-20; re-measured on every push by the image size step in .github/workflows/ci.yml

    701 MB at 0.1.3; the static ffmpeg layer is 104 MB. Reported by CI on every push, no longer gated

  • Runtime.create 23.5 ms

    verified

    evals/boot/results.json (createMs)

  • A remote provider's post-ready refresh takes ~1.5 s, so awaiting it in boot would multiply boot

    needs-run

    Decision 4.21 prose only (1,474 ms)

    No committed result

  • The gateway Dispach replaced blocked for minutes on network calls during hook initialisation

    needs-run

    None committed

    The README now says "observed, not measured" and gives no number; a figure needs the log evidence

  • Idle silo: 104 / 120 / 178 MiB RSS and 118 / 151 / 516 ms cold start for 1 / 10 / 100 agents; resume 5–6 ms

    verified

    evals/tenancy/results.json (2026-09-24)

    One machine

Prompting, context and memory

  • One emphatic framing line lifts qwen3.5:9b all-rules compliance by 20pp; deepseek-chat unchanged

    verified

    evals/prompt-style/results-intensity-6rules.json

    p≈0.16; the whole lift comes from one rule (digits)

  • Example placement (system vs user) makes no difference

    verified

    evals/prompt-style/results.json

    Saturated at 100% in both arms: "no difference at this difficulty", nothing more

  • Rule compliance: deepseek-v4-pro saturated at 64/64

    verified

    evals/rules/README.md

    Saturated

  • Rule compliance on a small model

    needs-run

    evals/rules/results.json holds a qwen3.5:9b run at 0.990; the README says no run finished

    Which is current is not recorded; the README now says so. Neither is quoted

  • Token-estimate correction: mean error 14.2% uncalibrated, 2.9% with the shipped EMA

    verified

    evals/budget/results.json

    One endpoint

  • A model writing a shell script into exec damages 25% of multi-line arguments, half silently

    verified

    evals/nlt-heredoc/results.json (2026-09-13)

    A floor on the problem, one endpoint; the backstop false-repairs about a third

  • Memory retrieval: median 0.34 ms, slowest 3.7 ms over 5,000 passages

    verified

    evals/memory/results.json

    One machine

  • Skill-index scan, 50 skills: cold well under the 50 ms criterion, warm under 5 ms

    verified

    evals/skills/results.json

    Cold varies with load (5–11 ms measured); quote the criterion, not a point

Safety and agents

Eval suites

  • bootBoot: process start to `runtime.ready`measured 2026-10-20
  • budgetThe estimator's bias, and the weight that corrects itmeasured date not recorded
  • handoffevals/handoff — does delegation cost the parent fewer tokens?no committed results
  • harnessHarness benchmarkmeasured 2026-10-23
  • memoryMemory retrievalmeasured 2026-10-20
  • nlt-heredocevals/nlt-heredoc — multi-line values through the NLT parsermeasured 2026-09-13
  • phasesPhase-scoped tool visibility, measured — and the number is negativemeasured date not recorded
  • prompt-stylepromptStyle — the two open questions, measuredmeasured 2026-08-14
  • rules`perRuleSuccess`, measuredmeasured date not recorded
  • scenariosScenario evals — layer 2measured 2026-09-27
  • skillsSkill indexingmeasured 2026-10-20
  • subagentsevals/subagents — is routing bulk tool work to a subagent worth it?measured 2026-10-03
  • tenancyDensity: what an idle silo costsmeasured 2026-09-24
  • toolsTool dialect comparison — NLT vs nativemeasured date not recorded
  • webevals/web — prompt injection through a fetched pagemeasured date not recorded