# Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving

arXiv: https://arxiv.org/abs/2512.12211 · HTML: https://mlsanity.com/paper/2512.12211/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 13 Dec 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 21 of 21)

- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction](https://mlsanity.com/paper/1903.07933/) (2019)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Scene-Graph Augmented Data-Driven Risk Assessment of Autonomous Vehicle Decisions](https://mlsanity.com/paper/2009.06435/) (2020)
- [Trajectory Prediction in Autonomous Driving with a Lane Heading Auxiliary Loss](https://mlsanity.com/paper/2011.06679/) (2020)
- [Path-Aware Graph Attention for HD Maps in Motion Prediction](https://mlsanity.com/paper/2202.13772/) (2022)
- [RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding](https://mlsanity.com/paper/2304.08600/) (2023)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification](https://mlsanity.com/paper/2404.18573/) (2024)
- [Scenario-based Evaluation of Prediction Models for Automated Vehicles](https://mlsanity.com/paper/2210.06553/) (2022)
- [Towards trustworthy multi-modal motion prediction: Holistic evaluation and interpretability of outputs](https://mlsanity.com/paper/2210.16144/) (2022)
- [Beelines: Motion Prediction Metrics for Self-Driving Safety and Comfort](https://mlsanity.com/paper/2011.00393/) (2020)
- [LMR: Lane Distance-Based Metric for Trajectory Prediction](https://mlsanity.com/paper/2304.05869/) (2023)
- [Gaussian Lane Keeping: A Robust Prediction Baseline](https://mlsanity.com/paper/2407.18451/) (2024)

## Abstract (arXiv)

Being able to anticipate the motion of surrounding agents is essential for the safe operation of autonomous driving systems in dynamic situations. While various methods have been proposed for trajectory prediction, the current evaluation practices still rely on error-based metrics (e.g., ADE, FDE), which reveal the accuracy from a post-hoc view but ignore the actual effect the predictor brings to the self-driving vehicles (SDVs), especially in complex interactive scenarios: a high-quality predictor not only chases accuracy, but should also captures all possible directions a neighbor agent might move, to support the SDVs' cautious decision-making. Given that the existing metrics hardly account for this standard, in our work, we propose a comprehensive pipeline that adaptively evaluates the predictor's performance by two dimensions: accuracy and diversity. Based on the criticality of the driving scenario, these two dimensions are dynamically combined and result in a final score for the predictor's performance. Extensive experiments on a closed-loop benchmark using real-world datasets show that our pipeline yields a more reasonable evaluation than traditional metrics by better reflecting the correlation of the predictors' evaluation with the autonomous vehicles' driving performance. This evaluation pipeline shows a robust way to select a predictor that potentially contributes most to the SDV's driving performance.