mlsanityData to 14 Oct 2026

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

arXiv 2610.04638 ↗First posted 3 Oct 2026cs.LG cs.AI math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity also fits: Deep RL algorithms and representations

Abstract (arXiv)

Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model Zθ to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-Q labels, RDA2C matches SAC at matched batch size and update frequency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 50 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

34 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces10 Mar
  2. 2025Mirror Descent Actor Critic via Bounded Advantage Learning6 Feb
  3. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr

Other references · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2018Maximum a Posteriori Policy Optimisationcited by 328
19 more
  1. 2019A Theory of Regularized Markov Decision Processescited by 286
  2. 2020Critic Regularized Regressioncited by 275
  3. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  4. 2017A unified view of entropy-regularized Markov decision processescited by 182
  5. 2020Revisiting Fundamentals of Experience Replaycited by 175
  6. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  7. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  8. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  9. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  10. 2020Mirror Descent Policy Optimizationcited by 106
  11. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  12. 2020Evaluating the Performance of Reinforcement Learning Algorithmscited by 36
  13. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
  14. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  15. 2022Policy Optimization over General State and Action Spacescited by 23
  16. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 18
  17. 2019Deep Conservative Policy Iterationcited by 15
  18. 2022Revisiting Discrete Soft Actor-Criticcited by 13
  19. 2024Dual Approximation Policy Optimizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
  2. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration
  3. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  4. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
  5. 2024vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvement
  6. 2025Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
  7. 2026Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
  8. 2026Variance Reduction Based Experience Replay for Policy Optimization