# Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Models

arXiv: https://arxiv.org/abs/2104.10558 · HTML: https://mlsanity.com/paper/2104.10558/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 21 of 21)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?](https://mlsanity.com/paper/2006.14911/) (2020)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [Deep Imitative Models for Flexible Inference, Planning, and Control](https://mlsanity.com/paper/1810.06544/) (2018)
- [Deep Generative Modeling of LiDAR Data](https://mlsanity.com/paper/1812.01180/) (2018)
- [Discrete Flows: Invertible Generative Models of Discrete Data](https://mlsanity.com/paper/1905.10347/) (2019)
- [Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1911.04175/) (2019)
- [Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction](https://mlsanity.com/paper/1710.09483/) (2017)
- [Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety](https://mlsanity.com/paper/1903.00640/) (2019)
- [Hierarchical Game-Theoretic Planning for Autonomous Vehicles](https://mlsanity.com/paper/1810.05766/) (2018)
- [Towards Learning Multi-agent Negotiations via Self-Play](https://mlsanity.com/paper/2001.10208/) (2020)
- [Analyzing Human Models that Adapt Online](https://mlsanity.com/paper/2103.05746/) (2021)

## Built on by (most cited first, 22 of 22)

- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning](https://mlsanity.com/paper/2401.00125/) (2023)
- [Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following](https://mlsanity.com/paper/2402.06559/) (2024)
- [Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?](https://mlsanity.com/paper/2404.07569/) (2024)
- [Autonomy 2.0: Why is self-driving always 5 years away?](https://mlsanity.com/paper/2107.08142/) (2021)
- [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning](https://mlsanity.com/paper/2111.11097/) (2021)
- [MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation](https://mlsanity.com/paper/2110.09481/) (2021)
- [QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2404.01486/) (2024)
- [Dream to Drive with Predictive Individual World Model](https://mlsanity.com/paper/2501.16733/) (2025)
- [Energy-based Potential Games for Joint Motion Forecasting and Control](https://mlsanity.com/paper/2312.01811/) (2023)
- [Fully Unified Motion Planning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2504.12667/) (2025)
- [CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting](https://mlsanity.com/paper/2401.18075/) (2024)
- [Dynamic High-Order Control Barrier Functions with Diffuser for Safety-Critical Trajectory Planning at Signal-Free Intersections](https://mlsanity.com/paper/2412.00162/) (2024)
- [Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving](https://mlsanity.com/paper/2510.15505/) (2025)
- [Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning](https://mlsanity.com/paper/2204.03597/) (2022)
- [HYPE: Hybrid Planning with Ego Proposal-Conditioned Predictions](https://mlsanity.com/paper/2510.12733/) (2025)
- [Driver Dojo: A Benchmark for Generalizable Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2207.11432/) (2022)
- [Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response](https://mlsanity.com/paper/2512.03936/) (2025)
- [Safer Autonomous Driving in a Stochastic, Partially-Observable Environment by Hierarchical Contingency Planning](https://mlsanity.com/paper/2204.06509/) (2022)
- [Tractable Joint Prediction and Planning over Discrete Behavior Modes for Urban Driving](https://mlsanity.com/paper/2403.07232/) (2024)

## Abstract (arXiv)

Humans have a remarkable ability to make decisions by accurately reasoning about future events, including the future behaviors and states of mind of other agents. Consider driving a car through a busy intersection: it is necessary to reason about the physics of the vehicle, the intentions of other drivers, and their beliefs about your own intentions. If you signal a turn, another driver might yield to you, or if you enter the passing lane, another driver might decelerate to give you room to merge in front. Competent drivers must plan how they can safely react to a variety of potential future behaviors of other agents before they make their next move. This requires contingency planning: explicitly planning a set of conditional actions that depend on the stochastic outcome of future events. In this work, we develop a general-purpose contingency planner that is learned end-to-end using high-dimensional scene observations and low-dimensional behavioral observations. We use a conditional autoregressive flow model to create a compact contingency planning space, and show how this model can tractably learn contingencies from behavioral observations. We developed a closed-loop control benchmark of realistic multi-agent scenarios in a driving simulator (CARLA), on which we compare our method to various noncontingent methods that reason about multi-agent future behavior, including several state-of-the-art deep learning-based planning approaches. We illustrate that these noncontingent planning methods fundamentally fail on this benchmark, and find that our deep contingency planning method achieves significantly superior performance. Code to run our benchmark and reproduce our results is available at https://sites.google.com/view/contingency-planning