Benchmark Experiment Template
Copy for every new benchmark. Change one variable at a time.
Copy to benchmarks/bNNN-slug.html, register it in site-data.js, and fill every field. Change exactly one row in the configuration table and mark it class="var-changed".
Question
- Benchmark name
- B-NNN · short name
- Research question
- one-sentence question
- Use case
- use case
- Dataset
- dataset, version, split
- Scale
- tasks × trials
- Baseline
- baseline
- Treatment
- treatment
- Changed variable
- exactly one
Configuration
| Variable | Baseline | Treatment |
|---|---|---|
| Harness | harness | harness |
| Model | model | model |
| Workflow | baseline workflow | treatment workflow |
| Agent configuration | agent config | agent config |
| Skill configuration | skills | skills |
| Policy configuration | policy | policy |
| Tool configuration | tools | tools |
| Rigor version | rigor | rigor |
Metrics
- Evaluation metrics
- how outputs are scored
- Outcome metrics
- list
- Process metrics
- list
- Latency
- definition
- Cost
- definition
- Reliability
- definition
- Human effort
- definition
Results
| Metric | Baseline | Treatment | Delta |
|---|---|---|---|
| Outcome metrics | |||
| Task success rate | — | — | — |
| Quality score (1–5) | — | — | — |
| Process metrics | |||
| Evidence completeness | — | — | — |
| Policy violations | — | — | — |
| Latency · cost · reliability · human effort | |||
| Wall-clock (min) | — | — | — |
| Cost (USD / task) | — | — | — |
| Run-to-run std. dev. (pp) | — | — | — |
| Human interventions | — | — | — |
Observations
what you saw
Failure modes
- failure mode
Limitations
- limitation
Conclusion
conclusion, scoped to the evidence
Next experiment
next one-variable change