mlsanityData to 14 Oct 2026

Equivalence Between Policy Gradients and Soft Q-Learning

arXiv 1704.06440 ↗First posted 21 Apr 2017cs.LGcited by 173

Not on the map. Why?

Cited by 34 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Two of the leading approaches for model-free reinforcement learning are policy gradient methods and Q-learning methods. Q-learning methods can be effective and sample-efficient when they work, however, it is not well-understood why they work, since empirically, the Q-values they estimate are very inaccurate. A partial explanation may be that Q-learning methods are secretly implementing policy gradient updates: we show that there is a precise equivalence between Q-learning and policy gradient methods in the setting of entropy-regularized reinforcement learning, that "soft" (entropy-regularized) Q-learning is exactly equivalent to a policy gradient method. We also point out a connection between Q-learning methods and natural policy gradient methods. Experimentally, we explore the entropy-regularized versions of Q-learning and policy gradients, and we find them to perform as well as (or slightly better than) the standard variants on the Atari benchmark. We also show that the equivalence holds in practical settings by constructing a Q-learning method that closely matches the learning dynamics of A3C without using a target network or ε-greedy exploration schedule.

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 3; 2021: 26; 2022: 25; 2023: 24; 2024: 34; 2025: 36; 2026: 24.

173 papers.

Sort:

By area:

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  3. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2018Self-Imitation Learningcited by 138
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  7. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  8. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 104
  9. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  10. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  11. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  12. 2022q-Learning in Continuous Timecited by 91
38 more
  1. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  2. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  3. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  4. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  5. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  6. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  7. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  8. 2022Generative Augmented Flow Networkscited by 50
  9. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 50
  10. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  11. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 44
  12. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  13. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 37
  14. 2021A New Formalism, Method and Open Issues for Zero-Shot Coordinationcited by 33
  15. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  16. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  17. 2021Regularized Policies are Reward Robustcited by 32
  18. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  19. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 29
  20. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 29
  21. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  22. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoningcited by 28
  23. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  24. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  25. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 22
  26. 2022Independent and Decentralized Learning in Markov Potential Gamescited by 22
  27. 2023Continuous-time q-learning for mean-field control problemscited by 22
  28. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 21
  29. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 20
  30. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyondcited by 20
  31. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
  32. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 19
  33. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  34. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  35. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 17
  36. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  37. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  38. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16

50 most cited of 173 All 173 (JSON)