# Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control

arXiv: https://arxiv.org/abs/1506.02632 · HTML: https://mlsanity.com/paper/1506.02632/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2015.

## Built on by (most cited first, 30 of 34)

- [Risk-Averse Offline Reinforcement Learning](https://mlsanity.com/paper/2102.05371/) (2021)
- [Off-Policy Risk Assessment in Contextual Bandits](https://mlsanity.com/paper/2104.08977/) (2021)
- [On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk](https://mlsanity.com/paper/2103.02827/) (2021)
- [RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training](https://mlsanity.com/paper/2510.00911/) (2025)
- [Quantile-Based Policy Optimization for Reinforcement Learning](https://mlsanity.com/paper/2201.11463/) (2022)
- [A Survey of Risk-Aware Multi-Armed Bandits](https://mlsanity.com/paper/2205.05843/) (2022)
- [Off-Policy Risk Assessment in Markov Decision Processes](https://mlsanity.com/paper/2209.10444/) (2022)
- [Modeling Recommender Ecosystems: Research Challenges at the Intersection of Mechanism Design, Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2309.06375/) (2023)
- [Policy Gradient Methods for Distortion Risk Measures](https://mlsanity.com/paper/2107.04422/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Supervised Learning with General Risk Functionals](https://mlsanity.com/paper/2206.13648/) (2022)
- [A Risk-Sensitive Approach to Policy Optimization](https://mlsanity.com/paper/2208.09106/) (2022)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning](https://mlsanity.com/paper/2410.04108/) (2024)
- [Online Estimation and Optimization of Utility-Based Shortfall Risk](https://mlsanity.com/paper/2111.08805/) (2021)
- [Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms](https://mlsanity.com/paper/2305.07248/) (2023)
- [Clustering Structure of Microstructure Measures](https://mlsanity.com/paper/2107.02283/) (2021)
- [A Black Swan Hypothesis: The Role of Human Irrationality in AI Safety](https://mlsanity.com/paper/2407.18422/) (2024)
- [Reinforcement Learning Beyond Expectation](https://mlsanity.com/paper/2104.00540/) (2021)
- [An Efficient Group-based Search Engine Marketing System for E-Commerce](https://mlsanity.com/paper/2106.12700/) (2021)
- [A policy gradient approach for optimization of smooth risk measures](https://mlsanity.com/paper/2202.11046/) (2022)
- [Deep Learning Based Page Creation for Improving E-Commerce Organic Search Traffic](https://mlsanity.com/paper/2209.10792/) (2022)
- [Concentration Bounds for Optimized Certainty Equivalent Risk Estimation](https://mlsanity.com/paper/2405.20933/) (2024)
- [Functional Linear Regression of Cumulative Distribution Functions](https://mlsanity.com/paper/2205.14545/) (2022)
- [Beyond Rational Illusion: Behaviorally Realistic Strategic Classification](https://mlsanity.com/paper/2605.19674/) (2026)
- [Joint Prediction of Remaining Useful Life and Failure Type of Train Wheelsets: A Multi-task Learning Approach](https://mlsanity.com/paper/2101.03497/) (2021)
- [Adversarial Driving Behavior Generation Incorporating Human Risk Cognition for Autonomous Vehicle Evaluation](https://mlsanity.com/paper/2310.00029/) (2023)
- [DRL-ORA: Distributional Reinforcement Learning with Online Risk Adaption](https://mlsanity.com/paper/2310.05179/) (2023)
- [Policy Newton methods for Distortion Riskmetrics](https://mlsanity.com/paper/2508.07249/) (2025)
- [Stochastic Approximation Methods for Distortion Risk Measure Optimization](https://mlsanity.com/paper/2510.04563/) (2025)

## Abstract (arXiv)

Cumulative prospect theory (CPT) is known to model human decisions well, with substantial empirical evidence supporting this claim. CPT works by distorting probabilities and is more general than the classic expected utility and coherent risk measures. We bring this idea to a risk-sensitive reinforcement learning (RL) setting and design algorithms for both estimation and control. The RL setting presents two particular challenges when CPT is applied: estimating the CPT objective requires estimations of the entire distribution of the value function and finding a randomized optimal policy. The estimation scheme that we propose uses the empirical distribution to estimate the CPT-value of a random variable. We then use this scheme in the inner loop of a CPT-value optimization procedure that is based on the well-known simulation optimization idea of simultaneous perturbation stochastic approximation (SPSA). We provide theoretical convergence guarantees for all the proposed algorithms and also illustrate the usefulness of CPT-based criteria in a traffic signal control application.