mlsanityData to 14 Oct 2026

Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization

arXiv 2304.00026 ↗First posted 31 Mar 2023cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper presents a review of the field of reinforcement learning (RL), with a focus on providing a comprehensive overview of the key concepts, techniques, and algorithms for beginners. RL has a unique setting, jargon, and mathematics that can be intimidating for those new to the field or artificial intelligence more broadly. While many papers review RL in the context of specific applications, such as games, healthcare, finance, or robotics, these papers can be difficult for beginners to follow due to the inclusion of non-RL-related work and the use of algorithms customized to those specific applications. To address these challenges, this paper provides a clear and concise overview of the fundamental principles of RL and covers the different types of RL algorithms. For each algorithm/method, we outline the main motivation behind its development, its inner workings, and its limitations. The presentation of the paper is aligned with the historical progress of the field, from the early 1980s Q-learning algorithm to the current state-of-the-art algorithms such as TD3, PPO, and offline RL. Overall, this paper aims to serve as a valuable resource for beginners looking to construct a solid understanding of the fundamentals of RL and be aware of the historical progress of the field. It is intended to be a go-to reference for those interested in learning about RL without being distracted by the details of specific applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 13 of its 32 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

13 of 32 references.

References · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  8. 2017Deep Reinforcement Learning: An Overviewcited by 333
  9. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  10. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  11. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  12. 2020Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economicscited by 23
1 more
  1. 2019The problem with DDPG: understanding failures in deterministic environments with sparse rewardscited by 20

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependenciescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  2. 2024Introduction to Reinforcement Learning
  3. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  4. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  5. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  6. 2025Reinforcement Learning from Human Feedback
  7. 2026Mathematical methods of reinforcement learning
  8. 2026Reinforcement Learning in Operational Research: A Technical Review and Practical Roadmap