Point-to-Pixel

Branch

On prod, YOLO vs HSV is selected with #define use_yolo. Some feature branches expose CMake POINT_TO_PIXEL_USE_YOLO.

Gap-fill (republishing unseen cones across short gaps) is on feature/perc/propagate_history. Prod history votes, associates, and orders only. See cone history.

point_to_pixel owns color (and short-term temporal association) for centroids that already exist on /cones. It does not invent new 3D cones from image boxes. Boxes without a projected LiDAR point are unused.

Position wrong → driver. Color or flicker wrong → here (or calibration / twist feeding this node).

Pipeline

1. /cones arrives with a LiDAR stamp
2. Nearest left/right frames are taken from camera deques
3. Optional motion model integrates /filter/twist across the stamp gap
4. Points project through projection_matrix_l/r
5. YOLO or HSV classifies each projected pixel; left/right merge by confidence
6. Cone history updates tracks and publishes /associated_colored_cones

Topics

Direction	Topic	Notes
In	/cones	Centroids (metres) from the driver
In	/filter/twist	Motion model and history propagation
In	camera frames	SeeCam default; ZED / RosCam via compile flags
Out	/associated_colored_cones	Planning (post-history)
Out	/colored_cones	Pre-history debug when publish_pre_cone_history_debug=true

Parameters (params.yaml)

Area	Keys
Extrinsics / lever	projection_matrix_l/r, imu_to_lidar
YOLO builds	model path, class ID map, confidence_threshold
HSV builds	*_filter_low/high
History	cone_hist_* association / lifetime / ordering
Debug	publish_pre_cone_history_debug

Matrix generation and math: physical calibration.

Failure modes

Symptom	Likely cause
Good /cones, systematic miss in image	Projection / calibration / wrong YAML
On cone but wrong class	Classifier, confidence, lighting; HSV ranges on HSV builds
One camera only wrong	That side’s matrix, undistort, right-cam $180^\circ$
Ok at rest, wrong moving	Twist deque, imu_to_lidar, motion model
Pre-history stable, post-history flickers	Cone history (or gap-fill gates on propagate_history)

Triage steps: ops. Stack context: architecture.

Build and run

colcon build --symlink-install --packages-select point_to_pixel
ros2 run point_to_pixel point_to_pixel --ros-args \
  --params-file src/point_to_pixel/config/params.yaml