# Offline Tracking with Object Permanence

arXiv: https://arxiv.org/abs/2310.01288 · HTML: https://mlsanity.com/paper/2310.01288/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [Unifying Voxel-based Representation with Transformer for 3D Object Detection](https://mlsanity.com/paper/2206.00630/) (2022)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [Neural Enhanced Belief Propagation on Factor Graphs](https://mlsanity.com/paper/2003.01998/) (2020)
- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [Auto4D: Learning to Label 4D Objects from Sequential Point Clouds](https://mlsanity.com/paper/2101.06586/) (2021)
- [Immortal Tracker: Tracklet Never Dies](https://mlsanity.com/paper/2111.13672/) (2021)
- [DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds](https://mlsanity.com/paper/2306.06023/) (2023)
- [Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection](https://mlsanity.com/paper/2304.12315/) (2023)
- [3D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attention](https://mlsanity.com/paper/2203.10926/) (2022)
- [ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking](https://mlsanity.com/paper/2211.03919/) (2022)
- [3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking](https://mlsanity.com/paper/2308.06635/) (2023)
- [Asymmetrical Bi-RNN for pedestrian trajectory encoding](https://mlsanity.com/paper/2106.04419/) (2021)

## Built on by (most cited first, 7 of 7)

- [LeAP: Consistent multi-domain 3D labeling using Foundation Models](https://mlsanity.com/paper/2502.03901/) (2025)
- [Bayesian Multiobject Tracking With Neural-Enhanced Motion and Measurement Models](https://mlsanity.com/paper/2506.18124/) (2025)
- [LiDAR MOT-DETR: A LiDAR-based Two-Stage Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2505.12753/) (2025)
- [nuScenes Revisited: Progress and Challenges in Autonomous Driving](https://mlsanity.com/paper/2512.02448/) (2025)
- [Offline-Poly: A Polyhedral Framework For Offline 3D Multi-Object Tracking](https://mlsanity.com/paper/2602.13772/) (2026)
- [4DRC-OCC: Robust Semantic Occupancy Prediction Through Fusion of 4D Radar and Camera](https://mlsanity.com/paper/2603.07794/) (2026)
- [MotionSync: Non-Causal Refinement of Causal Tracker for Label-Efficient 3D Perception](https://mlsanity.com/paper/2608.29567/) (2026)

## Abstract (arXiv)

To reduce the expensive labor cost for manual labeling autonomous driving datasets, an alternative is to automatically label the datasets using an offline perception system. However, objects might be temporally occluded. Such occlusion scenarios in the datasets are common yet underexplored in offline auto labeling. In this work, we propose an offline tracking model that focuses on occluded object tracks. It leverages the concept of object permanence which means objects continue to exist even if they are not observed anymore. The model contains three parts: a standard online tracker, a re-identification (Re-ID) module that associates tracklets before and after occlusion, and a track completion module that completes the fragmented tracks. The Re-ID module and the track completion module use the vectorized map as one of the inputs to refine the tracking results with occlusion. The model can effectively recover the occluded object trajectories. It achieves state-of-the-art performance in 3D multi-object tracking by significantly improving the original online tracking result, showing its potential to be applied in offline auto labeling as a useful plugin to improve tracking by recovering occlusions.