mlsanityData to 7 Oct 2026

UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning

arXiv 2111.11097 ↗First posted 22 Nov 2021cs.RO cs.AI cs.LGcited by 23

On the map: Offline RL and imitation learning from data → Imitation and offline RL for driving

Abstract (arXiv)

Offline reinforcement learning (RL) provides a framework for learning decision-making from offline data and therefore constitutes a promising approach for real-world applications as automated driving. Self-driving vehicles (SDV) learn a policy, which potentially even outperforms the behavior in the sub-optimal data set. Especially in safety-critical applications as automated driving, explainability and transferability are key to success. This motivates the use of model-based offline RL approaches, which leverage planning. However, current state-of-the-art methods often neglect the influence of aleatoric uncertainty arising from the stochastic behavior of multi-agent systems. This work proposes a novel approach for Uncertainty-aware Model-Based Offline REinforcement Learning Leveraging plAnning (UMBRELLA), which solves the prediction, planning, and control problem of the SDV jointly in an interpretable learning-based fashion. A trained action-conditioned stochastic dynamics model captures distinctively different future evolutions of the traffic scene. The analysis provides empirical evidence for the effectiveness of our approach in challenging automated driving simulations and based on a real-world public dataset.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 40 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

28 of 40 references.

References · 28

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  9. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  10. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  11. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
16 more
  1. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  2. 2019Causal Confusion in Imitation Learningcited by 286
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  5. 2020Model-Based Offline Planningcited by 126
  6. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  7. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  8. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  9. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 97
  10. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  11. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  12. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  13. 2021Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Modelscited by 22
  14. 2021Deep Structured Reactive Planningcited by 18
  15. 2020LUCIDGames: Online Unscented Inverse Dynamic Games for Adaptive Trajectory Prediction and Planningcited by 17
  16. 2019ALGAMES: A Fast Solver for Constrained Dynamic Gamescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 7; 2025: 4; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  2. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  5. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  6. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  7. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  8. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 13
  9. 2023RiskBench: A Scenario-based Benchmark for Risk Identificationcited by 12
  10. 2023Energy-based Potential Games for Joint Motion Forecasting and Controlcited by 9
  11. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  12. 2022Differentiable Constrained Imitation Learning for Robot Motion Planning and Controlcited by 7
11 more
  1. 2022State Advantage Weighting for Offline RLcited by 5
  2. 2024LoRD: Adapting Differentiable Driving Policies to Distribution Shiftscited by 4
  3. 2024Distilling LLMs' Decomposition Abilities into Compact Language Modelscited by 3
  4. 2025Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planningcited by 3
  5. 2024Disentangling Uncertainty for Safe Social Navigation using Deep Reinforcement Learningcited by 2
  6. 2024State-Constrained Offline Reinforcement Learningcited by 1
  7. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learningcited by 1
  8. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learningcited by 0
  9. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  10. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0
  11. 2026Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
  2. 2021Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving
  3. 2026DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving
  4. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving
  5. 2022Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios
  6. 2023Pay Attention to How You Drive: Safe and Adaptive Model-Based Reinforcement Learning for Off-Road Driving
  7. 2023Learning to Control Autonomous Fleets from Observation via Offline Reinforcement Learning
  8. 2022Optimizing Trajectories for Highway Driving with Offline Reinforcement Learning