# UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction

arXiv: https://arxiv.org/abs/2403.15098 · HTML: https://mlsanity.com/paper/2403.15098/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 64 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 30 of 47)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning](https://mlsanity.com/paper/2204.08499/) (2022)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling](https://mlsanity.com/paper/2306.12241/) (2023)
- [AdvDO: Realistic Adversarial Attacks for Trajectory Prediction](https://mlsanity.com/paper/2209.08744/) (2022)
- [On Exposing the Challenging Long Tail in Future Prediction of Traffic Actors](https://mlsanity.com/paper/2103.12474/) (2021)
- [Human Trajectory Prediction via Counterfactual Analysis](https://mlsanity.com/paper/2107.14202/) (2021)
- [Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective](https://mlsanity.com/paper/2111.14820/) (2021)
- [Vehicle trajectory prediction works, but not everywhere](https://mlsanity.com/paper/2112.03909/) (2021)
- [Are socially-aware trajectory prediction models really socially-aware?](https://mlsanity.com/paper/2108.10879/) (2021)
- [SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving](https://mlsanity.com/paper/2206.14116/) (2022)

## Built on by (most cited first, 30 of 52)

- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [GaussRender: Learning 3D Occupancy with Gaussian Rendering](https://mlsanity.com/paper/2502.05040/) (2025)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [TAROT: Targeted Data Selection via Optimal Transport](https://mlsanity.com/paper/2412.00420/) (2024)
- [TrajDiff: End-to-end Autonomous Driving without Perception Annotation](https://mlsanity.com/paper/2512.00723/) (2025)
- [MAD: Motion Appearance Decoupling for efficient Driving World Models](https://mlsanity.com/paper/2601.09452/) (2026)
- [Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecasting](https://mlsanity.com/paper/2407.09475/) (2024)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [Foresight in Motion: Reinforcing Trajectory Prediction with Reward Heuristics](https://mlsanity.com/paper/2507.12083/) (2025)
- [Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model](https://mlsanity.com/paper/2507.22615/) (2025)
- [Valeo4Cast: A Modular Approach to End-to-End Forecasting](https://mlsanity.com/paper/2406.08113/) (2024)
- [Unified Human Localization and Trajectory Prediction with Monocular Vision](https://mlsanity.com/paper/2503.03535/) (2025)
- [Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planning](https://mlsanity.com/paper/2507.04790/) (2025)
- [Motion Forecasting via Model-Based Risk Minimization](https://mlsanity.com/paper/2409.10585/) (2024)
- [Annealed Winner-Takes-All for Motion Forecasting](https://mlsanity.com/paper/2409.11172/) (2024)
- [LoRD: Adapting Differentiable Driving Policies to Distribution Shifts](https://mlsanity.com/paper/2410.09681/) (2024)
- [Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification](https://mlsanity.com/paper/2504.13111/) (2025)
- [A Generalizable Physics-Enhanced State Space Model for Long-Term Dynamics Forecasting in Complex Environments](https://mlsanity.com/paper/2507.10792/) (2025)
- [123D: Unifying Multi-Modal Autonomous Driving Data at Scale](https://mlsanity.com/paper/2605.08084/) (2026)
- [Toward Unified Practices in Trajectory Prediction Research on Bird's-Eye-View Datasets](https://mlsanity.com/paper/2405.00604/) (2024)
- [Traffic and Safety Rule Compliance of Humans in Diverse Driving Situations](https://mlsanity.com/paper/2411.01909/) (2024)
- [DECODE: Domain-aware Continual Domain Expansion for Motion Prediction](https://mlsanity.com/paper/2411.17917/) (2024)
- [PPT: Pretraining with Pseudo-Labeled Trajectories for Motion Forecasting](https://mlsanity.com/paper/2412.06491/) (2024)
- [AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reports](https://mlsanity.com/paper/2503.20654/) (2025)
- [RetroMotion: Retrocausal Motion Forecasting Models are Instructable](https://mlsanity.com/paper/2505.20414/) (2025)
- [Goal-based Trajectory Prediction for improved Cross-Dataset Generalization](https://mlsanity.com/paper/2507.18196/) (2025)
- [STEP: Structured Training and Evaluation Platform for benchmarking trajectory prediction models](https://mlsanity.com/paper/2509.14801/) (2025)
- [A Multi-Loss Strategy for Vehicle Trajectory Prediction: Combining Off-Road, Diversity, and Directional Consistency Losses](https://mlsanity.com/paper/2411.19747/) (2024)
- [AI-Powered CPS-Enabled Vulnerable-User-Aware Urban Transportation Digital Twin: Methods and Applications](https://mlsanity.com/paper/2501.10396/) (2024)
- [DISC: Dataset for Analyzing Driving Styles In Simulated Crashes for Mixed Autonomy](https://mlsanity.com/paper/2502.00050/) (2025)

## Abstract (arXiv)

Vehicle trajectory prediction has increasingly relied on data-driven solutions, but their ability to scale to different data domains and the impact of larger dataset sizes on their generalization remain under-explored. While these questions can be studied by employing multiple datasets, it is challenging due to several discrepancies, e.g., in data formats, map resolution, and semantic annotation types. To address these challenges, we introduce UniTraj, a comprehensive framework that unifies various datasets, models, and evaluation criteria, presenting new opportunities for the vehicle trajectory prediction field. In particular, using UniTraj, we conduct extensive experiments and find that model performance significantly drops when transferred to other datasets. However, enlarging data size and diversity can substantially improve performance, leading to a new state-of-the-art result for the nuScenes dataset. We provide insights into dataset characteristics to explain these findings. The code can be found here: https://github.com/vita-epfl/UniTraj