Evaluation Matrix

Metric catalogue used across benchmarks.

Metric families

Family	Metric	Definition	Unit	Direction
Outcome
Outcome	Task success rate	Share of tasks passing acceptance tests	%	↑
Outcome	Quality score	Rubric score from reviewer or judge	1–5	↑
Process
Process	Evidence completeness	Required evidence items present	%	↑
Process	Policy violations	Actions blocked or flagged by policy	count	↓
Efficiency
Latency	Wall-clock time	Start to accepted output	min	↓
Cost	Token / compute cost	Total spend per task	USD	↓
Reliability & effort
Reliability	Run-to-run variance	Std. dev. of success across trials	pp	↓
Human effort	Interventions	Human messages or corrections per task	count	↓
Human effort	Review time	Human minutes to accept output	min	↓