# Model-Free Risk-Sensitive Reinforcement Learning

arXiv: https://arxiv.org/abs/2111.02907 · HTML: https://mlsanity.com/paper/2111.02907/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 12 of 12)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [Meta-learning of Sequential Strategies](https://mlsanity.com/paper/1905.03030/) (2019)
- [Meta-trained agents implement Bayes-optimal agents](https://mlsanity.com/paper/2010.11223/) (2020)
- [Regularized Policies are Reward Robust](https://mlsanity.com/paper/2101.07012/) (2021)
- [If MaxEnt RL is the Answer, What is the Question?](https://mlsanity.com/paper/1910.01913/) (2019)
- [Exploration vs Exploitation vs Safety: Risk-averse Multi-Armed Bandits](https://mlsanity.com/paper/1401.1123/) (2014)
- [Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processes](https://mlsanity.com/paper/1604.02080/) (2016)
- [An Adversarial Interpretation of Information-Theoretic Bounded Rationality](https://mlsanity.com/paper/1404.5668/) (2014)

## Built on by (most cited first, 7 of 7)

- [Single-Trajectory Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2301.11721/) (2023)
- [Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping](https://mlsanity.com/paper/2209.07288/) (2022)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning](https://mlsanity.com/paper/2501.02087/) (2025)
- [Beyond Bayes-optimality: meta-learning what you know you don't know](https://mlsanity.com/paper/2209.15618/) (2022)
- [EduGym: An Environment and Notebook Suite for Reinforcement Learning Education](https://mlsanity.com/paper/2311.10590/) (2023)
- [Risk-Averse Reinforcement Learning with Itakura-Saito Loss](https://mlsanity.com/paper/2505.16925/) (2025)

## Abstract (arXiv)

We extend temporal-difference (TD) learning in order to obtain risk-sensitive, model-free reinforcement learning algorithms. This extension can be regarded as modification of the Rescorla-Wagner rule, where the (sigmoidal) stimulus is taken to be either the event of over- or underestimating the TD target. As a result, one obtains a stochastic approximation rule for estimating the free energy from i.i.d. samples generated by a Gaussian distribution with unknown mean and variance. Since the Gaussian free energy is known to be a certainty-equivalent sensitive to the mean and the variance, the learning rule has applications in risk-sensitive decision-making.