# Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection

arXiv: https://arxiv.org/abs/2304.12315 · HTML: https://mlsanity.com/paper/2304.12315/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 61 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 30 of 53)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [RangeDet:In Defense of Range View for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2103.10039/) (2021)
- [LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving](https://mlsanity.com/paper/1903.08701/) (2019)
- [DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets](https://mlsanity.com/paper/2301.06051/) (2023)
- [RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection](https://mlsanity.com/paper/2106.13365/) (2021)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [Annotating Object Instances with a Polygon-RNN](https://mlsanity.com/paper/1704.05548/) (2017)
- [AFDetV2: Rethinking the Necessity of the Second Stage for Object Detection from Point Clouds](https://mlsanity.com/paper/2112.09205/) (2021)
- [Fully Sparse 3D Object Detection](https://mlsanity.com/paper/2207.10035/) (2022)

## Built on by (most cited first, 28 of 28)

- [Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection](https://mlsanity.com/paper/2406.10700/) (2024)
- [FSD V2: Improving Fully Sparse 3D Object Detection with Virtual Voxels](https://mlsanity.com/paper/2308.03755/) (2023)
- [Perspective-Invariant 3D Object Detection](https://mlsanity.com/paper/2507.17665/) (2025)
- [MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving](https://mlsanity.com/paper/2409.16149/) (2024)
- [MS3D++: Ensemble of Experts for Multi-Source Unsupervised Domain Adaptation in 3D Object Detection](https://mlsanity.com/paper/2308.05988/) (2023)
- [Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving](https://mlsanity.com/paper/2403.08504/) (2024)
- [3EED: Ground Everything Everywhere in 3D](https://mlsanity.com/paper/2511.01755/) (2025)
- [MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2401.16305/) (2024)
- [RobMOT: Robust 3D Multi-Object Tracking by Observational Noise and State Estimation Drift Mitigation on LiDAR PointCloud](https://mlsanity.com/paper/2405.11536/) (2024)
- [SOAP: Cross-sensor Domain Adaptation for 3D Object Detection Using Stationary Object Aggregation Pseudo-labelling](https://mlsanity.com/paper/2401.04230/) (2024)
- [Offline Tracking with Object Permanence](https://mlsanity.com/paper/2310.01288/) (2023)
- [LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds](https://mlsanity.com/paper/2311.01444/) (2023)
- [Towards Stable 3D Object Detection](https://mlsanity.com/paper/2407.04305/) (2024)
- [Learning to Detect Objects from Multi-Agent LiDAR Scans without Manual Labels](https://mlsanity.com/paper/2503.08421/) (2025)
- [ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving](https://mlsanity.com/paper/2411.05311/) (2024)
- [Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection](https://mlsanity.com/paper/2412.05154/) (2024)
- [Learning better representations for crowded pedestrians in offboard LiDAR-camera 3D tracking-by-detection](https://mlsanity.com/paper/2505.16029/) (2025)
- [Offline-Poly: A Polyhedral Framework For Offline 3D Multi-Object Tracking](https://mlsanity.com/paper/2602.13772/) (2026)
- [Towards Accurate State Estimation: Motion Dynamics Kalman Filter for 3D Multi-Object Tracking](https://mlsanity.com/paper/2505.07254/) (2025)
- [M^3Detection: Multi-Frame Multi-Level Feature Fusion for Multi-Modal 3D Object Detection with Camera and 4D Imaging Radar](https://mlsanity.com/paper/2510.27166/) (2025)
- [RBF Weighted Hyper-Involution for RGB-D Object Detection](https://mlsanity.com/paper/2310.00342/) (2023)
- [RegTrack: Simplicity Beneath Complexity in Robust Multi-Modal 3D Multi-Object Tracking](https://mlsanity.com/paper/2409.00618/) (2024)
- [From Binary to Semantic: Utilizing Large-Scale Binary Occupancy Data for 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2507.13387/) (2025)
- [MEGA-PCC: A Mamba-based Efficient Approach for Joint Geometry and Attribute Point Cloud Compression](https://mlsanity.com/paper/2512.22463/) (2025)
- [UniDA3D: A Unified Domain-Adaptive Framework for Multi-View 3D Object Detection](https://mlsanity.com/paper/2603.27995/) (2026)
- [MotionSync: Non-Causal Refinement of Causal Tracker for Label-Efficient 3D Perception](https://mlsanity.com/paper/2608.29567/) (2026)
- [Do Spinning Radar Doppler Velocity Measurements Improve Vehicle Detection and Tracking?](https://mlsanity.com/paper/2609.21000/) (2026)
- [A Differentiable Optimization Framework for Registering Sequential Bounding Boxes with Point Cloud Stream](https://mlsanity.com/paper/2609.34103/) (2026)

## Abstract (arXiv)

This paper aims for high-performance offline LiDAR-based 3D object detection. We first observe that experienced human annotators annotate objects from a track-centric perspective. They first label the objects with clear shapes in a track, and then leverage the temporal coherence to infer the annotations of obscure objects. Drawing inspiration from this, we propose a high-performance offline detector in a track-centric perspective instead of the conventional object-centric perspective. Our method features a bidirectional tracking module and a track-centric learning module. Such a design allows our detector to infer and refine a complete track once the object is detected at a certain moment. We refer to this characteristic as "onCe detecTed, neveR Lost" and name the proposed system CTRL. Extensive experiments demonstrate the remarkable performance of our method, surpassing the human-level annotating accuracy and the previous state-of-the-art methods in the highly competitive Waymo Open Dataset without model ensemble. The code will be made publicly available at https://github.com/tusen-ai/SST.