Reference Comparison

A reference answers whether current code behaved consistently with the blessed run on the same test input. It is a drift baseline, not a claim of ground-truth correctness. If the recording bytes, perturbations, replay rate, or correction file changed, a metric delta is not a pure code regression.

References are keyed by (scenario, event, origin). origin is track_testing or sim. At most one file per triple; either origin may be absent. Each file pins one recording and selected metric values from one blessed run.

This is not a role: reference stream (vis/perc/GT cones that metrics score against). Those stay in the scenario YAML and are not replayed.

• regress run never writes a reference. It loads the reference, checks the current run is the same test, then compares metrics within scenario tolerances.
• regress reference update runs the scenario and writes new blessed values (reviewable JSON under references/<scenario>/<event>/<origin>.json).

Pinned inputs

A reference stores roughly:

• recording_id and content fingerprint
• SHA256 of each extracted input stream file
• Perturbation block from the manifest
• Replay config: rate, plus correction-file hash when configured

On compare:

Mismatch	Default behavior
Input stream hashes	Fail, unless --accept-input-hash-mismatch
Perturbations	Always fail
Replay config	Always fail

Metric tolerances

Only paths listed under reference.metrics in the scenario are checked. Each path may use:

• tolerance: absolute
• tolerance_percent: percent of blessed value
• direction: higher_is_better / lower_is_better: one-sided allowed drift
• No direction: symmetric band around the blessed value

Result categories

Outcome	Run status	Exit
Metrics outside tolerance	fail	1
Infra / readiness / extract failure	error	3 (metrics not trustworthy; comparison skipped)

Harness overview: regression harness. Operator notes: ops.