Projection Math

At runtime, point-to-pixel maps a LiDAR point $X = (x,y,z,1)^\top$ into image pixels with a single 3×4 matrix $P$:

$$\begin{aligned} p &= P X \\ u &= p_x / p_z \\ v &= p_y / p_z \end{aligned}$$

On the feature-branch calibrator, $P$ is not a free 12-DoF DLT matrix. It is composed from a known camera matrix and a rigid LiDAR→camera transform:

$$P = K_{\text{runtime}}\,[R \mid t]$$

where $[R \mid t]$ sends a point from the LiDAR frame into the OpenCV camera frame, and $K_{\text{runtime}}$ maps that 3D camera point into the runtime image that classifiers actually see.

Code: compose_projection in calib_geometry.py; application in state_manager.cpp::transform_points.

Frame axes

LiDAR points are stored as $x$ forward, $y$ left, $z$ up. OpenCV camera coordinates are $x$ right, $y$ down, $z$ forward. The extrinsic solve works in OpenCV axes; lidar_to_opencv_axes() is the fixed axis change used when seeding an initial $(R,t)$.

Runtime $K$ vs physical $K$

Intrinsics YAML stores the physical (pre-remap) camera matrix and distortion. SeeCam capture undistorts with getOptimalNewCameraMatrix(..., alpha=0), then the right camera rotates the image 180°:

$$(u,v) \mapsto (W-1-u,\; H-1-v)$$

The extrinsic solver therefore builds

$$K_{\text{runtime}} = \begin{cases} K_{\text{new}} & \text{left} \\ A\,K_{\text{new}} & \text{right} \end{cases} \quad A = \begin{bmatrix} -1 & 0 & W-1 \\ 0 & -1 & H-1 \\ 0 & 0 & 1 \end{bmatrix}$$

and writes $P = K_{\text{runtime}}[R \mid t]$ so that projecting into the already-rotated right image needs no extra pixel remap at solve time. apply_runtime_pixel_transform encodes the same 180° map if you start from a physical-pixel $P$.

If you calibrate against undistorted-but-not-rotated right frames, or feed distorted frames into the extrinsic scripts, the resulting $P$ will look systematically wrong even when $(R,t)$ is fine.

Reprojection error

For a LiDAR point on the board (or a cone), $(u,v)$ should match the corresponding image feature. The feature-branch write gate uses mean and p95 error between projected LiDAR pattern corners and detected chessboard corners (extrinsics math).

Prod click-DLT estimates $P$ directly from clicked pairs without forcing the $K[R|t]$ factorization; see legacy DLT.