# Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback

arXiv: https://arxiv.org/abs/2504.13785 · HTML: https://mlsanity.com/paper/2504.13785/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes](https://mlsanity.com/paper/2305.10430/) (2023)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving](https://mlsanity.com/paper/2408.00415/) (2024)
- [TPCN: Temporal Point Cloud Networks for Motion Forecasting](https://mlsanity.com/paper/2103.03067/) (2021)

## Built on by (most cited first, 4 of 4)

- [When Planners Meet Reality: How Learned, Reactive Traffic Agents Shift nuPlan Benchmarks](https://mlsanity.com/paper/2510.14677/) (2025)
- [Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models](https://mlsanity.com/paper/2602.04184/) (2026)
- [Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving](https://mlsanity.com/paper/2510.15505/) (2025)
- [DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion](https://mlsanity.com/paper/2506.12095/) (2025)

## Abstract (arXiv)

In automated driving, predicting trajectories of surrounding vehicles supports reasoning about scene dynamics and enables safe planning for the ego vehicle. However, existing models handle predictions as an instantaneous task of forecasting future trajectories based on observed information. As time proceeds, the next prediction is made independently of the previous one, which means that the model cannot correct its errors during inference and will repeat them. To alleviate this problem and better leverage temporal data, we propose a novel retrospection technique. Through training on closed-loop rollouts the model learns to use aggregated feedback. Given new observations it reflects on previous predictions and analyzes its errors to improve the quality of subsequent predictions. Thus, the model can learn to correct systematic errors during inference. Comprehensive experiments on nuScenes and Argoverse demonstrate a considerable decrease in minimum Average Displacement Error of up to 31.9% compared to the state-of-the-art baseline without retrospection. We further showcase the robustness of our technique by demonstrating a better handling of out-of-distribution scenarios with undetected road-users.