# Closing the Planning-Learning Loop with Application to Autonomous Driving

arXiv: https://arxiv.org/abs/2101.03834 · HTML: https://mlsanity.com/paper/2101.03834/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Model-Based Reinforcement Learning with Value-Targeted Regression](https://mlsanity.com/paper/2006.01107/) (2020)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Combining Optimal Control and Learning for Visual Navigation in Novel Environments](https://mlsanity.com/paper/1903.02531/) (2019)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic](https://mlsanity.com/paper/1911.04074/) (2019)
- [Learning to Drive from Simulation without Real World Labels](https://mlsanity.com/paper/1812.03823/) (2018)
- [PORCA: Modeling and Planning for Autonomous Driving among Many Pedestrians](https://mlsanity.com/paper/1805.11833/) (2018)
- [Learning to Search with MCTSnets](https://mlsanity.com/paper/1802.04697/) (2018)
- [Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation](https://mlsanity.com/paper/1910.13395/) (2019)
- [Deep Learning Alternative to Explicit Model Predictive Control for Unknown Nonlinear Systems](https://mlsanity.com/paper/2011.03699/) (2020)
- [HyP-DESPOT: A Hybrid Parallel Algorithm for Online Planning under Uncertainty](https://mlsanity.com/paper/1802.06215/) (2018)
- [MAGIC: Learning Macro-Actions for Online POMDP Planning](https://mlsanity.com/paper/2011.03813/) (2020)
- [LeTS-Drive: Driving in a Crowd by Learning from Tree Search](https://mlsanity.com/paper/1905.12197/) (2019)
- [GAMMA: A General Agent Motion Model for Autonomous Driving](https://mlsanity.com/paper/1906.01566/) (2019)

## Built on by (most cited first, 12 of 12)

- [DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.04049/) (2025)
- [BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations](https://mlsanity.com/paper/2306.00249/) (2023)
- [LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty](https://mlsanity.com/paper/2209.11422/) (2022)
- [World Engine: Towards the Era of Post-Training for Autonomous Driving](https://mlsanity.com/paper/2606.19836/) (2026)
- [Learning Online Belief Prediction for Efficient POMDP Planning in Autonomous Driving](https://mlsanity.com/paper/2401.15315/) (2024)
- [Learning Symbolic Persistent Macro-Actions for POMDP Solving Over Time](https://mlsanity.com/paper/2505.03668/) (2025)
- [Constrained Hierarchical Monte Carlo Belief-State Planning](https://mlsanity.com/paper/2310.20054/) (2023)
- [Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2406.13223/) (2024)
- [Zero-shot Transfer Learning of Driving Policy via Socially Adversarial Traffic Flow](https://mlsanity.com/paper/2304.12821/) (2023)
- [Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learning](https://mlsanity.com/paper/2509.17042/) (2025)
- [HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Driving](https://mlsanity.com/paper/2510.07210/) (2025)
- [GammaZero: Learning To Guide POMDP Belief Space Search With Graph Representations](https://mlsanity.com/paper/2510.14035/) (2025)

## Abstract (arXiv)

Real-time planning under uncertainty is critical for robots operating in complex dynamic environments. Consider, for example, an autonomous robot vehicle driving in dense, unregulated urban traffic of cars, motorcycles, buses, etc. The robot vehicle has to plan in both short and long terms, in order to interact with many traffic participants with uncertain intentions and drive effectively. Planning explicitly over a long time horizon, however, incurs prohibitive computational costs and is impractical under real-time constraints. To achieve real-time performance for large-scale planning, this work introduces a new algorithm Learning from Tree Search for Driving (LeTS-Drive), which integrates planning and learning in a closed loop, and applies it to autonomous driving in crowded urban traffic in simulation. Specifically, LeTS-Drive learns a policy and its value function from data provided by an online planner, which searches a sparsely-sampled belief tree; the online planner in turn uses the learned policy and value functions as heuristics to scale up its run-time performance for real-time robot control. These two steps are repeated to form a closed loop so that the planner and the learner inform each other and improve in synchrony. The algorithm learns on its own in a self-supervised manner, without human effort on explicit data labeling. Experimental results demonstrate that LeTS-Drive outperforms either planning or learning alone, as well as open-loop integration of planning and learning.