# Choquet regularization for reinforcement learning

arXiv: https://arxiv.org/abs/2208.08497 · HTML: https://mlsanity.com/paper/2208.08497/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 12 of 12)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations](https://mlsanity.com/paper/1505.05116/) (2015)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances](https://mlsanity.com/paper/1802.04885/) (2018)
- [State-Dependent Temperature Control for Langevin Diffusions](https://mlsanity.com/paper/2011.07456/) (2020)
- [Exploration versus exploitation in reinforcement learning: a stochastic control approach](https://mlsanity.com/paper/1812.01552/) (2018)
- [Robust exploratory mean-variance problem with drift uncertainty](https://mlsanity.com/paper/2108.04100/) (2021)
- [Risk averse non-stationary multi-armed bandits](https://mlsanity.com/paper/2109.13977/) (2021)

## Built on by (most cited first, 2 of 2)

- [Risk-sensitive Bandits: Arm Mixture Optimality and Regret-efficient Algorithms](https://mlsanity.com/paper/2503.08896/) (2025)
- [Policy Newton methods for Distortion Riskmetrics](https://mlsanity.com/paper/2508.07249/) (2025)

## Abstract (arXiv)

We propose Choquet regularizers to measure and manage the level of exploration for reinforcement learning (RL), and reformulate the continuous-time entropy-regularized RL problem of Wang et al. (2020, JMLR, 21(198)) in which we replace the differential entropy used for regularization with a Choquet regularizer. We derive the Hamilton--Jacobi--Bellman equation of the problem, and solve it explicitly in the linear--quadratic (LQ) case via maximizing statically a mean--variance constrained Choquet regularizer. Under the LQ setting, we derive explicit optimal distributions for several specific Choquet regularizers, and conversely identify the Choquet regularizers that generate a number of broadly used exploratory samplers such as ε-greedy, exponential, uniform and Gaussian.