mlsanityData to 7 Oct 2026

Divergence-Augmented Policy Optimization

arXiv 2501.15034 ↗First posted 25 Jan 2025cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 29 of its 56 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

29 of 56 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2016OpenAI Gymcited by 1,684
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2014Convex Optimization: Algorithms and Complexitycited by 569
17 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 318
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  3. 2018Distributed Prioritized Experience Replaycited by 260
  4. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  5. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  7. 2017A unified view of entropy-regularized Markov decision processescited by 174
  8. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  9. 2018Meta-Gradient Reinforcement Learningcited by 156
  10. 2018Self-Imitation Learningcited by 135
  11. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  12. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  13. 2017Boltzmann Exploration Done Rightcited by 69
  14. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  15. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18
  16. 2017f-Divergence constrained policy improvementcited by 15
  17. 2018Maximum Causal Tsallis Entropy Imitation Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  2. 2026GIPO: Gaussian Importance Sampling Policy Optimizationcited by 1
  3. 2025Mirror Descent Actor Critic via Bounded Advantage Learningcited by 0
  4. 2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0