mlsanityData to 14 Oct 2026

PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods

arXiv 2407.13146 ↗First posted 18 Jul 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper introduces PG-Rainbow, a novel algorithm that incorporates a distributional reinforcement learning framework with a policy gradient algorithm. Existing policy gradient methods are sample inefficient and rely on the mean of returns when calculating the state-action value function, neglecting the distributional nature of returns in reinforcement learning tasks. To address this issue, we use an Implicit Quantile Network that provides the quantile information of the distribution of rewards to the critic network of the Proximal Policy Optimization algorithm. We show empirical results that through the integration of reward distribution information into the policy network, the policy agent acquires enhanced capabilities to comprehensively evaluate the consequences of potential actions in a given state, facilitating more sophisticated and informed decision-making processes. We evaluate the performance of the proposed algorithm in the Atari-2600 game suite, simulated via the Arcade Learning Environment (ALE).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 20 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

14 of 20 references.

References · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  12. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
2 more
  1. 2020Phasic Policy Gradientcited by 123
  2. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Differential Perspective on Distributional Reinforcement Learning
  2. 2024Generalizing soft actor-critic algorithms to discrete action spaces
  3. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  4. 2021Policy Fusion for Adaptive and Customizable Reinforcement Learning Agents
  5. 2024Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks
  6. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
  7. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  8. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization