# Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes

arXiv: https://arxiv.org/abs/2104.03956 · HTML: https://mlsanity.com/paper/2104.03956/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Cost-Effective Active Learning for Deep Image Classification](https://mlsanity.com/paper/1701.03551/) (2017)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Contextual Diversity for Active Learning](https://mlsanity.com/paper/2008.05723/) (2020)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [Scalable Active Learning for Object Detection](https://mlsanity.com/paper/2004.04699/) (2020)
- [CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation](https://mlsanity.com/paper/1810.09726/) (2018)
- [Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector](https://mlsanity.com/paper/1901.10609/) (2019)
- [Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints](https://mlsanity.com/paper/1905.04753/) (2019)
- [Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction](https://mlsanity.com/paper/1910.08041/) (2019)
- [Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision](https://mlsanity.com/paper/2003.00393/) (2020)
- [Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs](https://mlsanity.com/paper/2011.06165/) (2020)
- [Diverse Complexity Measures for Dataset Curation in Self-driving](https://mlsanity.com/paper/2101.06554/) (2021)
- [HCNAF: Hyper-Conditioned Neural Autoregressive Flow and its Application for Probabilistic Occupancy Map Forecasting](https://mlsanity.com/paper/1912.08111/) (2019)

## Built on by (most cited first, 2 of 2)

- [Improving the Intra-class Long-tail in 3D Detection via Rare Example Mining](https://mlsanity.com/paper/2210.08375/) (2022)
- [BaSAL: Size-Balanced Warm Start Active Learning for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2310.08035/) (2023)

## Abstract (arXiv)

Self-driving vehicles must perceive and predict the future positions of nearby actors in order to avoid collisions and drive safely. A learned deep learning module is often responsible for this task, requiring large-scale, high-quality training datasets. As data collection is often significantly cheaper than labeling in this domain, the decision of which subset of examples to label can have a profound impact on model performance. Active learning techniques, which leverage the state of the current model to iteratively select examples for labeling, offer a promising solution to this problem. However, despite the appeal of this approach, there has been little scientific analysis of active learning approaches for the perception and prediction (P&P) problem. In this work, we study active learning techniques for P&P and find that the traditional active learning formulation is ill-suited for the P&P setting. We thus introduce generalizations that ensure that our approach is both cost-aware and allows for fine-grained selection of examples through partially labeled scenes. Our experiments on a real-world, large-scale self-driving dataset suggest that fine-grained selection can improve the performance across perception, prediction, and downstream planning tasks.