# Control-Aware Prediction Objectives for Autonomous Driving

arXiv: https://arxiv.org/abs/2204.13319 · HTML: https://mlsanity.com/paper/2204.13319/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [Model-Based Reinforcement Learning with Value-Targeted Regression](https://mlsanity.com/paper/2006.01107/) (2020)
- [Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations](https://mlsanity.com/paper/2008.05930/) (2020)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [Learning to Evaluate Perception Models Using Planner-Centric Metrics](https://mlsanity.com/paper/2004.08745/) (2020)
- [Goal-Aware Prediction: Learning to Model What Matters](https://mlsanity.com/paper/2007.07170/) (2020)
- [Social Attention for Autonomous Decision-Making in Dense Traffic](https://mlsanity.com/paper/1911.12250/) (2019)
- [Policy-Aware Model Learning for Policy Gradient Methods](https://mlsanity.com/paper/2003.00030/) (2020)
- [Goal-Driven Dynamics Learning via Bayesian Optimization](https://mlsanity.com/paper/1703.09260/) (2017)
- [Loss-Calibrated Approximate Inference in Bayesian Neural Networks](https://mlsanity.com/paper/1805.03901/) (2018)
- [Variational Bayesian Decision-making for Continuous Utilities](https://mlsanity.com/paper/1902.00792/) (2019)
- [Rethinking Trajectory Forecasting Evaluation](https://mlsanity.com/paper/2107.10297/) (2021)
- [Pragmatic Image Compression for Human-in-the-Loop Decision-Making](https://mlsanity.com/paper/2108.04219/) (2021)
- [Beelines: Motion Prediction Metrics for Self-Driving Safety and Comfort](https://mlsanity.com/paper/2011.00393/) (2020)

## Built on by (most cited first, 18 of 18)

- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [What Truly Matters in Trajectory Prediction for Autonomous Driving?](https://mlsanity.com/paper/2306.15136/) (2023)
- [AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Dataset](https://mlsanity.com/paper/2404.02429/) (2024)
- [Uncertainty estimation for Cross-dataset performance in Trajectory prediction](https://mlsanity.com/paper/2205.07310/) (2022)
- [RAP: Risk-Aware Prediction for Robust Planning](https://mlsanity.com/paper/2210.01368/) (2022)
- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformer](https://mlsanity.com/paper/2311.06623/) (2023)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives](https://mlsanity.com/paper/2505.09074/) (2025)
- [Risk-Driven Design of Perception Systems](https://mlsanity.com/paper/2205.10677/) (2022)
- [AVOIDDS: Aircraft Vision-based Intruder Detection Dataset and Simulator](https://mlsanity.com/paper/2306.11203/) (2023)
- [Quantifying Agent Interaction in Multi-agent Reinforcement Learning for Cost-efficient Generalization](https://mlsanity.com/paper/2310.07218/) (2023)
- [E-TIDE: Fast, Structure-Preserving Motion Forecasting from Event Sequences](https://mlsanity.com/paper/2603.27757/) (2026)
- [Visual Perception System for Autonomous Driving](https://mlsanity.com/paper/2303.02257/) (2023)
- [NashFormer: Leveraging Local Nash Equilibria for Semantically Diverse Trajectory Prediction](https://mlsanity.com/paper/2305.17600/) (2023)
- [Robust Planning for Autonomous Driving via Mixed Adversarial Diffusion Predictions](https://mlsanity.com/paper/2505.12327/) (2025)

## Abstract (arXiv)

Autonomous vehicle software is typically structured as a modular pipeline of individual components (e.g., perception, prediction, and planning) to help separate concerns into interpretable sub-tasks. Even when end-to-end training is possible, each module has its own set of objectives used for safety assurance, sample efficiency, regularization, or interpretability. However, intermediate objectives do not always align with overall system performance. For example, optimizing the likelihood of a trajectory prediction module might focus more on easy-to-predict agents than safety-critical or rare behaviors (e.g., jaywalking). In this paper, we present control-aware prediction objectives (CAPOs), to evaluate the downstream effect of predictions on control without requiring the planner be differentiable. We propose two types of importance weights that weight the predictive likelihood: one using an attention model between agents, and another based on control variation when exchanging predicted trajectories for ground truth trajectories. Experimentally, we show our objectives improve overall system performance in suburban driving scenarios using the CARLA simulator.