Beyond Bayes-optimality: meta-learning what you know you don't know
Not on the map. Why?
Abstract (arXiv)
Meta-training agents with memory has been shown to culminate in Bayes-optimal agents, which casts Bayes-optimality as the implicit solution to a numerical optimization problem rather than an explicit modeling assumption. Bayes-optimal agents are risk-neutral, since they solely attune to the expected return, and ambiguity-neutral, since they act in new situations as if the uncertainty were known. This is in contrast to risk-sensitive agents, which additionally exploit the higher-order moments of the return, and ambiguity-sensitive agents, which act differently when recognizing situations in which they lack knowledge. Humans are also known to be averse to ambiguity and sensitive to risk in ways that aren't Bayes-optimal, indicating that such sensitivity can confer advantages, especially in safety-critical situations. How can we extend the meta-learning protocol to generate risk- and ambiguity-sensitive agents? The goal of this work is to fill this gap in the literature by showing that risk- and ambiguity-sensitivity also emerge as the result of an optimization problem using modified meta-training algorithms, which manipulate the experience-generation process of the learner. We empirically test our proposed meta-training algorithms on agents exposed to foundational classes of decision-making experiments and demonstrate that they become sensitive to risk and ambiguity.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 73 references that are in the map. First posted on arXiv 30 Sep 2022.
Builds on
22 of 73 references.
References · 22
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2016Concrete Problems in AI Safetycited by 1,582
- 2016Deep Exploration via Bootstrapped DQNcited by 518
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
- 2016Learning to reinforcement learncited by 369
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2013Risk-sensitive Reinforcement Learningcited by 129
- 2021Epistemic Neural Networkscited by 120
- 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
- 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 86
10 more
- 2019Meta-learning of Sequential Strategiescited by 68
- 2021Risk-Averse Offline Reinforcement Learningcited by 65
- 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
- 2020Meta-trained agents implement Bayes-optimal agentscited by 37
- 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
- 2016Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processescited by 7
- 2020Semi-supervised novelty detection using ensembles with regularized disagreementcited by 7
- 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7
- 2017An Information-Theoretic Optimality Principle for Deep Reinforcement Learningcited by 3
- 2021Learning to Be Cautiouscited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.