# Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception

arXiv: https://arxiv.org/abs/2203.11405 · HTML: https://mlsanity.com/paper/2203.11405/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 43 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networks](https://mlsanity.com/paper/1609.06666/) (2016)
- [An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2007.12392/) (2020)
- [Driven to Distraction: Self-Supervised Distractor Learning for Robust Monocular Visual Odometry in Urban Environments](https://mlsanity.com/paper/1711.06623/) (2017)

## Built on by (most cited first, 20 of 20)

- [Neural Map Prior for Autonomous Driving](https://mlsanity.com/paper/2304.08481/) (2023)
- [Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions](https://mlsanity.com/paper/2208.01166/) (2022)
- [Enhancing Vectorized Map Perception with Historical Rasterized Maps](https://mlsanity.com/paper/2409.00620/) (2024)
- [PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors](https://mlsanity.com/paper/2403.09079/) (2024)
- [Unsupervised Adaptation from Repeated Traversals for Autonomous Driving](https://mlsanity.com/paper/2303.15286/) (2023)
- [Memorize What Matters: Emergent Scene Decomposition from Multitraverse](https://mlsanity.com/paper/2405.17187/) (2024)
- [3D Video Object Detection with Learnable Object-Centric Global Optimization](https://mlsanity.com/paper/2303.15416/) (2023)
- [Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene](https://mlsanity.com/paper/2502.06682/) (2025)
- [Better Monocular 3D Detectors with LiDAR from the Past](https://mlsanity.com/paper/2404.05139/) (2024)
- [Learning 3D Perception from Others' Predictions](https://mlsanity.com/paper/2410.02646/) (2024)
- [HyperMODEST: Self-Supervised 3D Object Detection with Confidence Score Filtering](https://mlsanity.com/paper/2304.14446/) (2023)
- [Leveraging Previous-Traversal Point Cloud Map Priors for Camera-Based 3D Object Detection and Tracking](https://mlsanity.com/paper/2604.25405/) (2026)
- [DisPlacing Objects: Improving Dynamic Vehicle Detection via Visual Place Recognition under Adverse Conditions](https://mlsanity.com/paper/2306.17536/) (2023)
- [Compressed Map Priors for 3D Perception](https://mlsanity.com/paper/2601.00139/) (2025)
- [Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features](https://mlsanity.com/paper/2309.12140/) (2023)
- [CSMapping: Scalable Crowdsourced Semantic Mapping and Topology Inference for Autonomous Driving](https://mlsanity.com/paper/2512.03510/) (2025)
- [When the City Teaches the Car: Label-Free 3D Perception from Infrastructure](https://mlsanity.com/paper/2603.16742/) (2026)
- [Scene Reconstruction as Mapping Priors for 3D Detection](https://mlsanity.com/paper/2605.22997/) (2026)
- [Driving on Memory](https://mlsanity.com/paper/2608.31029/) (2026)
- [Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping](https://mlsanity.com/paper/2609.26325/) (2026)

## Abstract (arXiv)

Self-driving cars must detect vehicles, pedestrians, and other traffic participants accurately to operate safely. Small, far-away, or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them. To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the same scene. We posit that these past data, which are typically discarded, provide rich contextual information for disambiguating the above-mentioned challenging cases. To this end, we propose a novel, end-to-end trainable Hindsight framework to extract this contextual information from past traversals and store it in an easy-to-query data structure, which can then be leveraged to aid future 3D object detection of the same scene. We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision on multiple autonomous driving datasets, most notably by more than 300% on the challenging cases.