B-000 · Vanilla Claude Code vs Claude Code + MDLC workflow
Filled-in example of the experiment template, using invented numbers.
All numbers on this page are invented to demonstrate the layout. They are not measured results.
Question
- Benchmark name
- B-000 · Vanilla Claude Code vs Claude Code + MDLC workflow
- Research question
- Does adding a structured MDLC workflow improve outcome quality enough to justify extra time and cost?
- Use case
- Build and validate a tabular classification model from a written requirement
- Dataset
- Public tabular dataset (to be chosen); fixed train/test split
- Scale
- 20 tasks × 5 trials per arm
- Baseline
- Vanilla Claude Code
- Treatment
- Claude Code + MDLC workflow
- Changed variable
- Workflow
Configuration
| Variable | Baseline | Treatment |
|---|---|---|
| Harness | Claude Code | Claude Code |
| Model | Sonnet | Sonnet |
| Workflow | Vanilla | MDLC workflow |
| Agent configuration | default | default |
| Skill configuration | none | none |
| Policy configuration | default permissions | default permissions |
| Tool configuration | built-in tools | built-in tools |
| Rigor version | v1 | v1 |
Metrics
- Evaluation metrics
- Acceptance tests on held-out data; reviewer rubric
- Outcome metrics
- Task success, quality score
- Process metrics
- Evidence completeness, policy violations
- Latency
- Wall-clock to accepted output
- Cost
- Token cost per task
- Reliability
- Std. dev. of success across trials
- Human effort
- Interventions per task
Results
| Metric | Baseline | Treatment | Delta |
|---|---|---|---|
| Outcome metrics | |||
| Task success rate | 60% | 75% | +15 pp |
| Quality score (1–5) | 3.4 | 3.9 | +0.5 |
| Process metrics | |||
| Evidence completeness | 40% | 90% | +50 pp |
| Policy violations | 2 | 2 | 0 |
| Latency · cost · reliability · human effort | |||
| Wall-clock (min) | 18 | 26 | +8 |
| Cost (USD / task) | 1.10 | 1.60 | +0.50 |
| Run-to-run std. dev. (pp) | 12 | 7 | −5 |
| Human interventions | 4.0 | 2.5 | −1.5 |
Observations
Treatment runs spent most of the extra time writing evaluation evidence before declaring completion.
Failure modes
- Both arms occasionally over-fit to the validation split when not told it was held out.
Limitations
- Single dataset; small n; one model.
Conclusion
If real, a +15 pp success gain at +45% cost would favour the workflow for high-stakes models only.
Next experiment
B-001: same setup, change Model (Sonnet → Opus) under the MDLC workflow.