mlsanityData to 7 Oct 2026

Towards Optimal Policy Improvement

arXiv 2610.01566 ↗First posted 1 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Practical Reinforcement Learning (RL) algorithms learn to solve Markov Decision Processes (MDPs) through iterative policy improvement in the presence of approximate evaluation. We study policy improvement from first principles, defining optimal policy improvement as producing the best policy attainable in a single update under specified constraints. We show that optimal improvement restricted to a set of states is equivalent to solving an induced MDP, characterizing planning with an explicit or implicit model as a path towards optimal policy improvement. Because practical methods commonly solve such induced problems through iterative improvement in the form of greedification, we take steps towards optimal greedification under the central practical constraint of approximate evaluation. We formulate greedification under this constraint as probabilistic decision-making under uncertainty and derive a novel operator that is optimal with respect to the resulting objective. Empirically, the operator and its practical gradient-based approximations improve aggregate performance across GumbelAlphaZero, SAC, ReBRAC and Generalized Policy Iteration, in experiments spanning discrete and continuous actions, model-based and model-free, online and offline RL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 72 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

36 of 72 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2023Mastering Diverse Domains through World Modelscited by 1,193
  9. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  10. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
24 more
  1. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  3. 2018Maximum a Posteriori Policy Optimisationcited by 318
  4. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  5. 2017A unified view of entropy-regularized Markov decision processescited by 174
  6. 2013A General Framework for Updating Belief Distributionscited by 172
  7. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  9. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  10. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
  11. 2020Mirror Descent Policy Optimizationcited by 102
  12. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 59
  13. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 45
  14. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  15. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  16. 2020Adaptive Sampling for Best Policy Identification in Markov Decision Processescited by 20
  17. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  18. 2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
  19. 2022Epistemic Monte Carlo Tree Searchcited by 7
  20. 2020Uncertainty-Aware Policy Optimization: A Robust, Adaptive Trust Region Approachcited by 6
  21. 2024Value Improved Actor Critic Algorithmscited by 6
  22. 2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
  23. 2025Trust-Region Twisted Policy Improvementcited by 5
  24. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy Improvement Reinforcement Learning
  2. 2025Generalization in Monitored Markov Decision Processes (Mon-MDPs)
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  4. 2023Safe Policy Improvement for POMDPs via Finite-State Controllers
  5. 2026Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success
  6. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
  7. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  8. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints