Regression Harness

The harness replays a bag through the live driver and point-to-pixel (plus any scenario nodes), scores named metrics, and compares them to a blessed reference for a (scenario, event, origin) triple. Use it to catch perceptions regressions on a known recording after a code change.

The pipeline under test is the same ROS graph you run on the car, or a controlled subset declared in the scenario. It is not a unit test of one function.

Piece	Owns
`harness_engine`	Discovery, extraction cache, orchestration, metrics, references, CLI
`bag_extractor` / `replay_node`	One timestamp-ordered timeline: LiDAR packets into the driver, passthrough of other streams

bag → extract → manifest → replay_node
    → lidar_driver → /cones (+ passthrough camera/twist/…)
    → point_to_pixel → capture → metrics ↔ reference

Stages

regress run executes:

Load and validate the scenario YAML
Resolve recording (event/origin reference pin, --recording, or --cmp)
Extract required streams into .regression_cache/ (or reuse cache)
Write replay manifest
Wait for graph readiness (wait_for_publisher / subscriber gates)
Replay on one interleaved timeline (replay timeline)
Collect configured output topics
Score metrics; compare to reference unless --cmp (reference comparison)

Exit codes

Code	Meaning
0	Pass
1	Metric / reference drift (capture looked valid; behavior moved)
2	Scenario or config error
3	Infrastructure / data / readiness failure (do not trust metrics)

Artifacts

Event runs: test_runs/<scenario>/<event>/<origin>/<timestamp>/
Ad-hoc: test_runs/<scenario>/cmp/<recording>/<timestamp>/

Each run directory includes run.json (status, metrics, reference result), replay_summary.json, and logs/.

References

A reference is a drift baseline, not ground truth. Distinct from a role: reference stream (vis/perc/GT cones that metrics score against).

Keyed by (scenario, event, origin). origin is track_testing or sim. At most one blessed recording per triple; either origin may be absent.
Written only by regress reference update, never by run. Files live under references/<scenario>/<event>/<origin>.json.
Pin recording fingerprint, input stream hashes, perturbations, and replay config (including rate / correction hash when set).
Metric paths and tolerances live in the scenario YAML under reference.metrics.

A metric delta after an intentional code change is a re-bless, not a silent edit of run.json. Inspect run.json before updating a reference. Details: reference comparison.

Data root and recordings

Resolution order: --data-root, then $REGRESS_DATA_ROOT, then /root/rosbags.

A recording is a ROS 2 bag (metadata.yaml) or a pre-extracted stream set (cache manifest.json). Optional recording.yaml sidecar tags event (e.g. skidpad) and origin (track_testing or sim) for discovery.

--cmp <id|dir|path> runs a throwaway comparison: no reference read/write. Use it to probe a recording without blessing.

Failure mapping

Observation	Area
Fails before `replay_summary.json`	Extract, stream format, manifest, `replay_node` startup / readiness
Empty `/cones` during replay	Replay path, correction `.dat`, driver params, packet stream
Cones ok, colors bad	point-to-pixel, camera/twist passthrough, fake camera, projection
Exit 1, infra clean	Metric drift vs reference. Read `run.json` before `reference update`
Input-hash / perturbation / replay-config mismatch	Different test than the reference pins; not a pure code regression

Commands

colcon build --symlink-install --packages-select lidar_driver bag_extractor harness_engine
source /opt/ros/humble/setup.bash && source install/setup.bash
pip install -e src/harness_engine

regress list --scenario cone_history_synthcam --data-root /root/rosbags
regress validate
regress reference update cone_history_synthcam --event skidpad --origin track_testing --recording <id>
regress run cone_history_synthcam --event skidpad --origin track_testing
regress run smoke_replay_only --cmp <recording> --data-root /root/rosbags --yes

list / validate / unit tests under harness_engine/test need little or no live ROS; full run needs the overlay.

ADRs under harness_engine in the driverless tree. Operator triage: ops.