mlsanityData to 7 Oct 2026

Risk-sensitive Reinforcement Learning

arXiv 1311.2097 ↗First posted 8 Nov 2013cs.LGcited by 129

Not on the map. Why?

Abstract (arXiv)

We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents' behaviors express key features of human behavior as predicted by prospect theory (Kahneman and Tversky, 1979), for example different risk-preferences for gains and losses as well as the shape of subjective probability curves. We derive a risk-sensitive Q-learning algorithm, which is necessary for modeling human behavior when transition probabilities are unknown, and prove its convergence. As a proof of principle for the applicability of the new framework we apply it to quantify human behavior in a sequential investment task. We find, that the risk-sensitive variant provides a significantly better fit to the behavioral data and that it leads to an interpretation of the subject's responses which is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals show a significant correlation of the risk-sensitive TD error with BOLD signal change in the ventral striatum. In addition we find a significant correlation of the risk-sensitive Q-values with neural activity in the striatum, cingulate cortex and insula, which is not present if standard Q-values are used.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 24; 2023: 22; 2024: 20; 2025: 11; 2026: 23.

129 papers.

Sort:

By area:

  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  2. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  3. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  4. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  5. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  6. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  7. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  8. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  9. 2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 22
  10. 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
  11. 2021Deep Reinforcement Learning for Active High Frequency Tradingcited by 21
  12. 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 20
38 more
  1. 2021Robust Risk-Aware Reinforcement Learningcited by 17
  2. 2022Learn to Match with No Regret: Reinforcement Learning in Markov Matching Marketscited by 16
  3. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  4. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  5. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  6. 2021Convergence and Alignment of Gradient Descent with Random Backpropagation Weightscited by 14
  7. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 14
  8. 2024Three Dogmas of Reinforcement Learningcited by 14
  9. 2021Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithmcited by 12
  10. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  11. 2021Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal controlcited by 12
  12. 2022RAP: Risk-Aware Prediction for Robust Planningcited by 12
  13. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  14. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
  15. 2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11
  16. 2023Beyond Average Return in Markov Decision Processescited by 11
  17. 2024The Evolution of Reinforcement Learning in Quantitative Finance: A Surveycited by 11
  18. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  19. 2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
  20. 2021Safe Adaptive Learning-based Control for Constrained Linear Quadratic Regulators with Regret Guaranteescited by 10
  21. 2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 9
  22. 2022Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learningcited by 9
  23. 2023Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functionscited by 9
  24. 2024What type of inference is planning?cited by 9
  25. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
  26. 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7
  27. 2023Modeling Recommender Ecosystems: Research Challenges at the Intersection of Mechanism Design, Reinforcement Learning and Generative Modelscited by 7
  28. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7
  29. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  30. 2021Distributional Offline Continuous-Time Reinforcement Learning with Neural Physics-Informed PDEs (SciPhy RL for DOCTR-L)cited by 6
  31. 2021Policy Gradient Methods for Distortion Risk Measurescited by 6
  32. 2022Risk-Sensitive Policy with Distributional Reinforcement Learningcited by 6
  33. 2023Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learningcited by 6
  34. 2023Learning to Forecast Aleatoric and Epistemic Uncertainties over Long Horizon Trajectoriescited by 6
  35. 2023Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Drivingcited by 6
  36. 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysiscited by 6
  37. 2021Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Searchcited by 5
  38. 2022Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertaintycited by 5

50 most cited of 129 All 129 (JSON)