mlsanityData to 7 Oct 2026

Challenging Common Assumptions in Convex Reinforcement Learning

arXiv 2202.01511 ↗First posted 3 Feb 2022cs.LGcited by 29

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The classic Reinforcement Learning (RL) formulation concerns the maximization of a scalar reward function. More recently, convex RL has been introduced to extend the RL formulation to all the objectives that are convex functions of the state distribution induced by a policy. Notably, convex RL covers several relevant applications that do not fall into the scalar formulation, including imitation learning, risk-averse RL, and pure exploration. In classic RL, it is common to optimize an infinite trials objective, which accounts for the state distribution instead of the empirical state visitation frequencies, even though the actual number of trajectories is always finite in practice. This is theoretically sound since the infinite trials and finite trials objectives can be proved to coincide and thus lead to the same optimal policy. In this paper, we show that this hidden assumption does not hold in the convex RL setting. In particular, we show that erroneously optimizing the infinite trials objective in place of the actual finite trials one, as it is usually done, can lead to a significant approximation error. Since the finite trials setting is the default in both simulated and real-world RL, we believe shedding light on this issue will lead to better approaches and methodologies for convex RL, impacting relevant research areas such as imitation learning, risk-averse RL, and pure exploration among others.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 75 references that are in the map. First posted on arXiv 3 Feb 2022.

Builds on

40 of 75 references.

References · 40

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  7. 2016Variational Intrinsic Controlcited by 262
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  9. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  10. 2019Efficient Exploration via State Marginal Matchingcited by 186
  11. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  12. 2021Reinforcement Learning with Prototypical Representationscited by 179
28 more
  1. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  2. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  3. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  4. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  5. 2020Primal Wasserstein Imitation Learningcited by 120
  6. 2021APS: Active Pretraining with Successor Featurescited by 109
  7. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  8. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  9. 2021Reward is enough for convex MDPscited by 82
  10. 2015Policy Gradient for Coherent Risk Measurescited by 74
  11. 2021On the Expressivity of Markov Rewardcited by 71
  12. 2019Reinforcement Learning with Convex Constraintscited by 57
  13. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  14. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 53
  15. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
  16. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  17. 2020Constrained episodic reinforcement learning in concave-convex and knapsack settingscited by 33
  18. 2021Geometric Entropic Explorationcited by 32
  19. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  20. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  21. 2019Active Exploration in Markov Decision Processescited by 27
  22. 2021Density Constrained Reinforcement Learningcited by 25
  23. 2021Wasserstein Unsupervised Reinforcement Learningcited by 21
  24. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  25. 2021Provably Efficient Algorithms for Multi-Objective Competitive RLcited by 20
  26. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  27. 2013Variance Adjusted Actor Critic Algorithmscited by 10
  28. 2019Exploration-Exploitation Trade-off in Reinforcement Learning on Online Markov Decision Processes with Global Concave Rewardscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 8; 2025: 6; 2026: 8.

29 papers.

Sort:

By area:

  1. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  2. 2023Submodular Reinforcement Learningcited by 23
  3. 2024Learning diffusion at lightspeedcited by 21
  4. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  5. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 17
  6. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  7. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15
  8. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 15
  9. 2024Three Dogmas of Reinforcement Learningcited by 14
  10. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  11. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  12. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
17 more
  1. 2022Policy Gradient for Reinforcement Learning with General Utilitiescited by 6
  2. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  3. 2024Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstractioncited by 6
  4. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learningcited by 6
  5. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 6
  6. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 6
  7. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  8. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  9. 2024Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learningcited by 3
  10. 2025Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learningcited by 2
  11. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 2
  12. 2026A Unified Density Operator View of Flow Control and Mergingcited by 2
  13. 2026Active Inference as a Convex Markov Decision Processcited by 1
  14. 2025Online Episodic Convex Reinforcement Learningcited by 0
  15. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0
  16. 2026Theoretical Foundations of max@k Reinforcement Learningcited by 0
  17. 2026The Dually Flat Geometry of Planning as Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  2. 2026Maximum Likelihood Reinforcement Learning
  3. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
  4. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  5. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
  6. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs
  7. 2023Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learning
  8. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review