Learning to Be Cautious
Not on the map. Why?
Abstract (arXiv)
A key challenge in the field of reinforcement learning is to develop agents that behave cautiously in novel situations. It is generally impossible to anticipate all situations that an autonomous system may face or what behavior would best avoid bad outcomes. An agent that can learn to be cautious would overcome this challenge by discovering for itself when and how to behave cautiously. In contrast, current approaches typically embed task-specific safety information or explicit cautious behaviors into the system, which is error-prone and imposes extra burdens on practitioners. In this paper, we present both a sequence of tasks where cautious behavior becomes increasingly non-obvious, as well as an algorithm to demonstrate that it is possible for a system to learn to be cautious. The essential features of our algorithm are that it characterizes reward function uncertainty without task-specific safety information and uses this uncertainty to construct a robust policy. Specifically, we construct robust policies with a k-of-N counterfactual regret minimization (CFR) subroutine given learned reward function uncertainty represented by a neural network ensemble. These policies exhibit caution in each of our tasks without any task-specific safety tuning. Our code is available at https://github.com/montaserFath/Learning-to-be-Cautious
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 29 Oct 2021.
Builds on
39 of 72 references.
References · 39
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2017Constrained Policy Optimizationcited by 928
- 2019On the Convergence of Adam and Beyondcited by 834
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
- 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
- 2017Noisy Networks for Explorationcited by 262
- 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
27 more
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2017Deep Exploration via Randomized Value Functionscited by 155
- 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
- 2018Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifierscited by 126
- 2021Epistemic Neural Networkscited by 120
- 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
- 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
- 2017AI Safety Gridworldscited by 108
- 2018Deep Counterfactual Regret Minimizationcited by 102
- 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 86
- 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
- 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
- 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
- 2017Ensemble Samplingcited by 62
- 2019Worst Cases Policy Gradientscited by 52
- 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
- 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
- 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
- 2019Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descentcited by 31
- 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 28
- 2020Hindsight and Sequential Rationality of Correlated Playcited by 21
- 2021Discovering a set of policies for the worst case rewardcited by 21
- 2020Stochastic Regret Minimization in Extensive-Form Gamescited by 17
- 2014Solving Games with Functional Regret Estimationcited by 12
- 2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11
- 2019Alternative Function Approximation Parameterizations for Solving Games: An Analysis of f-Regression Counterfactual Regret Minimizationcited by 5
- 2018Bayesian Neural Network Ensemblescited by 3
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1; 2026: 1.
3 papers.