mlsanityData to 7 Oct 2026

Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios

arXiv 2204.05196 ↗First posted 11 Apr 2022cs.LGcited by 0

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

When learning to behave in a stochastic environment where safety is critical, such as driving a vehicle in traffic, it is natural for human drivers to plan fallback strategies as a backup to use if ever there is an unexpected change in the environment. Knowing to expect the unexpected, and planning for such outcomes, increases our capability for being robust to unseen scenarios and may help prevent catastrophic failures. Control of Autonomous Vehicles (AVs) has a particular interest in knowing when and how to use fallback strategies in the interest of safety. Due to imperfect information available to an AV about its environment, it is important to have alternate strategies at the ready which might not have been deduced from the original training data distribution. In this paper we present a principled approach for a model-free Reinforcement Learning (RL) agent to capture multiple modes of behaviour in an environment. We introduce an extra pseudo-reward term to the reward model, to encourage exploration to areas of state-space different from areas privileged by the optimal policy. We base this reward term on a distance metric between the trajectories of agents, in order to force policies to focus on different areas of state-space than the initial exploring agent. Throughout the paper, we refer to this particular training paradigm as learning fallback strategies. We apply this method to an autonomous driving scenario, and show that we are able to learn useful policies that would have otherwise been missed out on during training, and unavailable to use when executing the control algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 18 of its 26 references that are in the map. First posted on arXiv 11 Apr 2022.

Builds on

18 of 26 references.

References · 18

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2017Hindsight Experience Replaycited by 1,096
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2015Prioritized Experience Replaycited by 1,070
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  7. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  8. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  9. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  10. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  11. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  12. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
6 more
  1. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  2. 2019Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Drivingcited by 52
  3. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  4. 2019Safe Reinforcement Learning with Scene Decomposition for Navigating Complex Urban Environmentscited by 18
  5. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
  6. 2020Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Building Safer Autonomous Agents by Leveraging Risky Driving Behavior Knowledge
  2. 2022How to Learn from Risk: Explicit Risk-Utility Reinforcement Learning for Efficient and Safe Driving Strategies
  3. 2025Balancing Progress and Safety: A Novel Risk-Aware Objective for RL in Autonomous Driving
  4. 2022Safer Autonomous Driving in a Stochastic, Partially-Observable Environment by Hierarchical Contingency Planning
  5. 2023Risk-Aware Reward Shaping of Reinforcement Learning Agents for Autonomous Driving
  6. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
  7. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving
  8. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning