mlsanityData to 7 Oct 2026

A Theory of Regularized Markov Decision Processes

arXiv 1901.11275 ↗First posted 31 Jan 2019cs.LG stat.MLcited by 279

Not on the map. Why?

Cited by 42 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes that generalizes these approaches in two directions: we consider a larger class of regularizers, and we consider the general modified policy iteration approach, encompassing both policy iteration and value iteration. The core building blocks of this theory are a notion of regularized Bellman operator and the Legendre-Fenchel transform, a classical tool of convex optimization. This approach allows for error propagation analyses of general algorithmic schemes of which (possibly variants of) classical algorithms such as Trust Region Policy Optimization, Soft Q-learning, Stochastic Actor Critic or Dynamic Policy Programming are special cases. This also draws connections to proximal convex optimization, especially to Mirror Descent.

Key facts. Cited by 279 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 4; 2021: 44; 2022: 36; 2023: 38; 2024: 47; 2025: 55; 2026: 55.

279 papers.

Sort:

By area:

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  4. 2023Nash Learning from Human Feedbackcited by 220
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  6. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  7. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  9. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  10. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  11. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  12. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
38 more
  1. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  2. 2021Reward is enough for convex MDPscited by 82
  3. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  4. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  5. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  6. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  7. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  8. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 61
  9. 2023Generative Flow Networks as Entropy-Regularized RLcited by 60
  10. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  11. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  12. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  13. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  14. 2021Adversarially Guided Actor-Criticcited by 47
  15. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  16. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  17. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  18. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  19. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  20. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  21. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  22. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
  23. 2021Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithmcited by 39
  24. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  25. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  26. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  27. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  28. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
  29. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  30. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  31. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  32. 2021Online Apprenticeship Learningcited by 28
  33. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  34. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  35. 2022Proximal Point Imitation Learningcited by 25
  36. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 25
  37. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 24
  38. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24

50 most cited of 279 All 279 (JSON)