# A Differentiable Optimization Framework for Registering Sequential Bounding Boxes with Point Cloud Stream

arXiv: https://arxiv.org/abs/2609.34103 · HTML: https://mlsanity.com/paper/2609.34103/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration](https://mlsanity.com/paper/1605.03344/) (2016)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [Auto4D: Learning to Label 4D Objects from Sequential Point Clouds](https://mlsanity.com/paper/2101.06586/) (2021)
- [ICP-Flow: LiDAR Scene Flow Estimation with ICP](https://mlsanity.com/paper/2402.17351/) (2024)
- [Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection](https://mlsanity.com/paper/2304.12315/) (2023)
- [Neural Eulerian Scene Flow Fields](https://mlsanity.com/paper/2410.02031/) (2024)
- [PointCaM: Cut-and-Mix for Open-Set Point Cloud Learning](https://mlsanity.com/paper/2212.02011/) (2022)
- [Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Prior](https://mlsanity.com/paper/2507.06651/) (2025)
- [Efficient and accurate object detection with simultaneous classification and tracking](https://mlsanity.com/paper/2007.02065/) (2020)
- [ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving](https://mlsanity.com/paper/2411.05311/) (2024)
- [OpenBox: Annotate Any Bounding Boxes in 3D](https://mlsanity.com/paper/2512.01352/) (2025)
- [Real-time 3D object proposal generation and classification under limited processing resources](https://mlsanity.com/paper/2003.10670/) (2020)
- [DC-Reg: Globally Optimal Point Cloud Registration via Tight Bounding with Difference of Convex Programming](https://mlsanity.com/paper/2603.25442/) (2026)

## Abstract (arXiv)

Refining a sequence of coarse 3D bounding boxes against a LiDAR point-cloud stream demands tracks that are geometrically accurate (high IoU) and temporally coherent (low roughness), preferably without training data. The usual recipe keeps the two concerns apart: register each frame independently, then smooth the trajectory afterwards with a Kalman~RTS or Savitzky--Golay filter. Smoothing displaces boxes from a geometric optimum and never re-optimises, so it trades accuracy for smoothness. We instead fold the temporal smoothness constraint into a training-free registration objective and solve for all poses jointly with L-BFGS. The payoff depends on how well the object is seen. On well-observed tracks it is large: within the low-roughness budget, the joint objective beats both post-hoc smoothers on paired multi-seed statistics and cuts roughness several-fold relative to frame-wise registration at matched accuracy. Treating visibility as an experimental variable exposes the limit. The advantage decays monotonically as views become one-sided, until it is indistinguishable from zero for near-edge-on objects and slightly negative under a ray-cast simulator with range-dependent density and ego motion, where the decoupled pipeline is in fact ahead at tight roughness budgets. We locate that boundary and trace it to one term: orientation alignment ties yaw to the estimated velocity and fails once that estimate is noisy. A ground-truth-free rule can choose the temporal scale and keep every track inside the roughness budget.