Benchmarking Framework

How experiments are designed so that each one changes exactly one variable.

Principle

Every benchmark compares a baseline and a treatment that differ in exactly one variable. Everything else is held constant and recorded.

Benchmarking system: one task set, two arms, one changed variable.
Fig. 1Benchmarking system: one task set, two arms, one changed variable.Open SVG ↗

Controllable variables

VariableExample valuesTypical question
HarnessClaude Code · Codex · otherDoes the harness matter for the same model?
ModelSonnet · OpusIs the larger model worth the cost?
WorkflowVanilla · MDLC workflowDoes a structured workflow improve outcomes?
Rigor versionRigor v1 · Rigor v2Did the new rigor level help?
Skillsnone · skill set ADo reusable skills reduce effort?
Policypermissive · strictWhat does stricter policy cost?
ToolsMCP set A · set BWhich tools help?

Protocol

  1. State the research question and the single variable being changed.
  2. Freeze the task set, seeds and all other configuration; record them in the experiment page.
  3. Run n trials per arm (report n).
  4. Score with the metrics in the evaluation matrix.
  5. Report deltas against baseline, with variance, failure modes and limitations.
  6. Propose the next experiment.

If more than one variable changed, the result is labelled exploratory and is not added to the results register as a controlled comparison.

Naming

Experiments are numbered B-001, B-002, … and saved as benchmarks/b001-short-slug.html from the experiment template.