# Policy Gradients for Cumulative Prospect Theory in Reinforcement Learning

arXiv: https://arxiv.org/abs/2410.02605 · HTML: https://mlsanity.com/paper/2410.02605/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 3 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Abstract (arXiv)

We derive a policy gradient theorem for Cumulative Prospect Theory (CPT) objectives in finite-horizon Reinforcement Learning (RL), generalizing the standard policy gradient theorem and encompassing distortion-based risk objectives as special cases. Motivated by behavioral economics, CPT combines an asymmetric utility transformation around a reference point with probability distortion. Building on our theorem, we design a first-order policy gradient algorithm for CPT-RL using a Monte Carlo gradient estimator based on order statistics. We establish statistical guarantees for the estimator and prove asymptotic convergence of the resulting algorithm to first-order stationary points of the (generally nonconvex) CPT objective. We complement our asymptotic analysis with a non-asymptotic total sample complexity analysis to reach an approximate first-order stationary policy. Simulations illustrate qualitative behaviors induced by CPT and compare our first-order approach to existing zeroth-order methods.