Benchmarks
Dispach is measured on 25 multi-turn agent tasks, 5 times each, against the
same tools and pinned model providers as two other open-source harnesses. Each task is graded on the tools'
final state, never on the reply's wording. Every run is committed, with its replies, under
evals/harness/.
Against other harnesses
Tasks passed out of 125, and what the 125 runs cost
on OpenRouter. Dispach uses native tool calls. Dispach was measured at 4f3e00a;
Hermes 0.19.0 on 2026-10-22 to 2026-10-23 and OpenClaw 2026.10.5 on
2026-10-22 to 2026-10-23, with the same tasks, isolation and providers.
| Model | Dispach | Hermes 0.19.0 | OpenClaw 2026.10.5 |
|---|---|---|---|
| Claude Opus 5.5 | 125/125 (100%) · $2.05 | 119/125 (95%) · $2.24 | 105/125 (84%) · $7.10 |
| Claude Sonnet 5.5 | 125/125 (100%) · $0.80 | 120/125 (96%) · $1.76 | 105/125 (84%) · $4.21 |
| Claude Haiku 5.5 | 124/125 (99%) · $0.06 | 120/125 (96%) · $0.07 | 102/125 (82%) · $0.21 |
| GPT-6.1 Sol | 125/125 (100%) · $0.54 | 120/125 (96%) · $0.53 | 95/125 (76%) · $1.73 |
| Gemini 4 Argon | 119/125 (95%) · $2.29 | 123/125 (98%) · $2.79 | 108/125 (86%) · $5.47 |
| GLM 5.3 | 125/125 (100%) · $0.80 | 123/125 (98%) · $1.18 | 98/125 (78%) · $4.86 |
| DeepSeek V4 Pro | 125/125 (100%) · $0.31 | 120/125 (96%) · $0.50 | 96/125 (77%) · $2.83 |
| Kimi K3 | 115/125 (92%) · $2.24 | 115/125 (92%) · $2.36 | 98/125 (78%) · $10.27 |
| Gemma 4 26B-A4B | 112/125 (90%) · $0.24 | 108/125 (86%) · $0.41 | 91/125 (73%) · $2.95 |
| qwen3.5-9b | 104/125 (83%) · $0.18 | 112/125 (90%) · $0.42 | 60/125 (48%) · $0.78 |
| gpt-oss-20b | 97/125 (78%) · $0.05 | 111/125 (89%) · $0.10 | 51/125 (41%) · $0.12 |
With 125 runs an arm, a gap of a few runs is noise: the 95% intervals are in
evals/harness/summary.json.
Tool-call dialects
Dispach can call tools through the provider's native function calling or through NLT, a plain-text
format. init writes native wherever the model supports it.
| Model | Native | NLT |
|---|---|---|
| Claude Opus 5.5 | 125/125 (100%) | 125/125 (100%) |
| Claude Sonnet 5.5 | 125/125 (100%) | 124/125 (99%) |
| Claude Haiku 5.5 | 124/125 (99%) | 124/125 (99%) |
| GPT-6.1 Sol | 125/125 (100%) | 125/125 (100%) |
| GPT-6 Luna | 114/125 (91%) | 85/125 (68%) |
| Gemini 4 Argon | 119/125 (95%) | 118/125 (94%) |
| Gemini 3.8 Flash | 125/125 (100%) | 119/125 (95%) |
| Grok 4.7 | 124/125 (99%) | 62/125 (50%) |
| GLM 5.3 | 125/125 (100%) | 119/125 (95%) |
| DeepSeek V4 Pro | 125/125 (100%) | 124/125 (99%) |
| Kimi K3 | 115/125 (92%) | 71/125 (57%) |
| Gemma 4 26B-A4B | 112/125 (90%) | 75/125 (60%) |
| qwen3.5-9b | 104/125 (83%) | paused at 45/52 |
| gpt-oss-20b | 97/125 (78%) | 89/125 (71%) |
How it is run
- Tasks: single calls, chains, recall across turns, abstention, recovery from injected tool failures,
and long files (
scripts/bench/tasks.ts). - Isolation: every harness runs in its own container, its state restored before every attempt.
- Scoring: a run that errors scores 0. A run the provider rate-limited is re-run, not scored.
- Limits: one benchmark of our own design, each harness on its own defaults. Where Dispach loses, and why, is written up in the benchmark README.