# On Exposing the Challenging Long Tail in Future Prediction of Traffic Actors

arXiv: https://arxiv.org/abs/2103.12474 · HTML: https://mlsanity.com/paper/2103.12474/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 80 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data](https://mlsanity.com/paper/1508.03422/) (2015)
- [Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax](https://mlsanity.com/paper/2006.10408/) (2020)
- [Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective](https://mlsanity.com/paper/2003.10780/) (2020)
- [M2m: Imbalanced Classification via Major-to-minor Translation](https://mlsanity.com/paper/2004.00431/) (2020)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Range Loss for Deep Face Recognition with Long-tail](https://mlsanity.com/paper/1611.08976/) (2016)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)

## Built on by (most cited first, 30 of 48)

- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving](https://mlsanity.com/paper/2408.10845/) (2024)
- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [Adaptive Trajectory Prediction via Transferable GNN](https://mlsanity.com/paper/2203.05046/) (2022)
- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [Social-Implicit: Rethinking Trajectory Prediction Evaluation and The Effectiveness of Implicit Maximum Likelihood Estimation](https://mlsanity.com/paper/2203.03057/) (2022)
- [FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Prediction](https://mlsanity.com/paper/2303.16574/) (2023)
- [You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction](https://mlsanity.com/paper/2110.05304/) (2021)
- [Long-Tail Prediction Uncertainty Aware Trajectory Planning for Self-driving Vehicles](https://mlsanity.com/paper/2207.00788/) (2022)
- [CausalAgents: A Robustness Benchmark for Motion Forecasting using Causal Relationships](https://mlsanity.com/paper/2207.03586/) (2022)
- [Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer](https://mlsanity.com/paper/2310.07587/) (2023)
- [Safety-Critical Scenario Generation Via Reinforcement Learning Based Editing](https://mlsanity.com/paper/2306.14131/) (2023)
- [Action-based Contrastive Learning for Trajectory Prediction](https://mlsanity.com/paper/2207.08664/) (2022)
- [Holistic Representation Learning for Multitask Trajectory Anomaly Detection](https://mlsanity.com/paper/2311.01851/) (2023)
- [Generating Out-Of-Distribution Scenarios Using Language Models](https://mlsanity.com/paper/2411.16554/) (2024)
- [Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models](https://mlsanity.com/paper/2310.08577/) (2023)
- [Vision-based Multi-future Trajectory Prediction: A Survey](https://mlsanity.com/paper/2302.10463/) (2023)
- [Taming the Long Tail of Deep Probabilistic Forecasting](https://mlsanity.com/paper/2202.13418/) (2022)
- [TrACT: A Training Dynamics Aware Contrastive Learning Framework for Long-tail Trajectory Prediction](https://mlsanity.com/paper/2404.12538/) (2024)
- [Raising context awareness in motion forecasting](https://mlsanity.com/paper/2109.08048/) (2021)
- [Multi-Agent Trajectory Prediction with Difficulty-Guided Feature Enhancement Network](https://mlsanity.com/paper/2407.18551/) (2024)
- [Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model](https://mlsanity.com/paper/2507.22615/) (2025)
- [CC-SGG: Corner Case Scenario Generation using Learned Scene Graphs](https://mlsanity.com/paper/2309.09844/) (2023)
- [AMEND: A Mixture of Experts Framework for Long-tailed Trajectory Prediction](https://mlsanity.com/paper/2402.08698/) (2024)
- [AMD: Adaptive Momentum and Decoupled Contrastive Learning Framework for Robust Long-Tail Trajectory Prediction](https://mlsanity.com/paper/2507.01801/) (2025)
- [Joint Forecasting of Panoptic Segmentations with Difference Attention](https://mlsanity.com/paper/2204.07157/) (2022)
- [G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction](https://mlsanity.com/paper/2404.19330/) (2024)
- [Den-TP: A Density-Balanced Data Curation and Evaluation Framework for Trajectory Prediction](https://mlsanity.com/paper/2409.17385/) (2024)

## Abstract (arXiv)

Predicting the states of dynamic traffic actors into the future is important for autonomous systems to operate safelyand efficiently. Remarkably, the most critical scenarios aremuch less frequent and more complex than the uncriticalones. Therefore, uncritical cases dominate the prediction. In this paper, we address specifically the challenging scenarios at the long tail of the dataset distribution. Our analysis shows that the common losses tend to place challenging cases suboptimally in the embedding space. As a consequence, we propose to supplement the usual loss with aloss that places challenging cases closer to each other. This triggers sharing information among challenging cases andlearning specific predictive features. We show on four public datasets that this leads to improved performance on the challenging scenarios while the overall performance stays stable. The approach is agnostic w.r.t. the used network architecture, input modality or viewpoint, and can be integrated into existing solutions easily. Code is available at https://github.com/lmb-freiburg/Contrastive-Future-Trajectory-Prediction