# Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula

arXiv: https://arxiv.org/abs/2212.01375 · HTML: https://mlsanity.com/paper/2212.01375/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](https://mlsanity.com/paper/1810.00825/) (2018)
- [When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?](https://mlsanity.com/paper/2204.05618/) (2022)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021)
- [CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1810.06284/) (2018)
- [Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model](https://mlsanity.com/paper/1810.12836/) (2018)
- [Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving](https://mlsanity.com/paper/2210.09539/) (2022)
- [Variance reduction properties of the reparameterization trick](https://mlsanity.com/paper/1809.10330/) (2018)
- [Fingerprint Policy Optimisation for Robust Reinforcement Learning](https://mlsanity.com/paper/1805.10662/) (2018)

## Built on by (most cited first, 17 of 17)

- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [Learning Realistic Traffic Agents in Closed-loop](https://mlsanity.com/paper/2311.01394/) (2023)
- [CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectories](https://mlsanity.com/paper/2403.13208/) (2024)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)
- [LoRD: Adapting Differentiable Driving Policies to Distribution Shifts](https://mlsanity.com/paper/2410.09681/) (2024)
- [Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling](https://mlsanity.com/paper/2411.17826/) (2024)
- [V-Max: A Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2503.08388/) (2025)
- [Scaling Is All You Need: Autonomous Driving with JAX-Accelerated Reinforcement Learning](https://mlsanity.com/paper/2312.15122/) (2023)
- [AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent](https://mlsanity.com/paper/2404.07428/) (2024)
- [Synthetic Data for Robust AI Model Development in Regulated Enterprises](https://mlsanity.com/paper/2503.12353/) (2025)
- [Foundation Models for Rapid Autonomy Validation](https://mlsanity.com/paper/2411.03328/) (2024)
- [Systems Engineering for Autonomous Vehicles; Supervising AI using Large Language Models (SSuperLLM)](https://mlsanity.com/paper/2501.10839/) (2025)
- [What should an AI assessor optimise for?](https://mlsanity.com/paper/2502.00365/) (2025)
- [CAPS: Context-Aware Priority Sampling for Enhanced Imitation Learning in Autonomous Driving](https://mlsanity.com/paper/2503.01650/) (2025)
- [Learning to Drive by Imitating Surrounding Vehicles](https://mlsanity.com/paper/2503.05997/) (2025)
- [From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Driving](https://mlsanity.com/paper/2606.14032/) (2026)
- [Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Driving](https://mlsanity.com/paper/2607.10438/) (2026)

## Abstract (arXiv)

ML-based motion planning is a promising approach to produce agents that exhibit complex behaviors, and automatically adapt to novel environments. In the context of autonomous driving, it is common to treat all available training data equally. However, this approach produces agents that do not perform robustly in safety-critical settings, an issue that cannot be addressed by simply adding more data to the training set - we show that an agent trained using only a 10% subset of the data performs just as well as an agent trained on the entire dataset. We present a method to predict the inherent difficulty of a driving situation given data collected from a fleet of autonomous vehicles deployed on public roads. We then demonstrate that this difficulty score can be used in a zero-shot transfer to generate curricula for an imitation-learning based planning agent. Compared to training on the entire unbiased training dataset, we show that prioritizing difficult driving scenarios both reduces collisions by 15% and increases route adherence by 14% in closed-loop evaluation, all while using only 10% of the training data.