mlsanityData to 7 Oct 2026

Momentum in Reinforcement Learning

arXiv 1910.09322 ↗First posted 21 Oct 2019cs.LG stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive q-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates this momentum idea. Our analysis shows that this allows MoVI to average errors over successive iterations. We show that the proposed approach can be readily extended to deep learning. Specifically, we propose a simple improvement on DQN based on MoVI, and experiment it on Atari games.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 6; 2022: 5; 2023: 2; 2024: 4; 2025: 0; 2026: 1.

19 papers.

Sort:

By area:

  1. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 19
  2. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  3. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  4. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  5. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 11
  6. 2021Optimization Issues in KL-Constrained Approximate Policy Iterationcited by 10
  7. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  8. 2023Accelerating Value Iteration with Anchoringcited by 10
  9. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  10. 2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
  11. 2022Learning to Constrain Policy Optimization with Virtual Trust Regioncited by 4
  12. 2024Deflated Dynamics Value Iterationcited by 4
7 more
  1. 2024Functional Acceleration for Policy Mirror Descentcited by 3
  2. 2021Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learningcited by 1
  3. 2024PID Accelerated Temporal Difference Algorithmscited by 1
  4. 2021Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learningcited by 0
  5. 2022q-Munchausen Reinforcement Learningcited by 0
  6. 2022A Parametric Class of Approximate Gradient Updates for Policy Optimizationcited by 0
  7. 2023Acceleration in Policy Optimizationcited by 0