mlsanityData to 7 Oct 2026

GMAC: A Distributional Perspective on Actor-Critic Framework

arXiv 2105.11366 ↗First posted 24 May 2021cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In this paper, we devise a distributional framework on actor-critic as a solution to distributional instability, action type restriction, and conflation between samples and statistics. We propose a new method that minimizes the Cram'er distance with the multi-step Bellman target distribution generated from a novel Sample-Replacement algorithm denoted SR(λ), which learns the correct value distribution under multiple Bellman operations. Parameterizing a value distribution with Gaussian Mixture Model further improves the efficiency and the performance of the method, which we name GMAC. We empirically show that GMAC captures the correct representation of value distributions and improves the performance of a conventional actor-critic method with low computational cost, in both discrete and continuous action spaces using Arcade Learning Environment (ALE) and PyBullet environment.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

23 of 53 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2017Wasserstein GANcited by 635
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
11 more
  1. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  2. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 116
  3. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  4. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  5. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  6. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  7. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
  8. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  9. 2019Distributional reinforcement learning with linear function approximationcited by 15
  10. 2020Sample-based Distributional Policy Gradientcited by 12
  11. 2018Nonlinear Distributional Gradient Temporal-Difference Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 4; 2024: 3; 2025: 0; 2026: 2.

15 papers.

Sort:

  1. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  2. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  3. 2023Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learningcited by 12
  4. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  5. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  6. 2022Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approachcited by 6
  7. 2022Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Taskscited by 4
  8. 2022Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Controlcited by 3
  9. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  10. 2024A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learningcited by 1
  11. 2024Off-policy Distributional Q(λ): Distributional RL without Importance Samplingcited by 1
  12. 2022Invariance to Quantile Selection in Distributional Continuous Controlcited by 0
3 more
  1. 2024Combining AI Control Systems and Human Decision Support via Robustness and Criticalitycited by 0
  2. 2026Distributional Active Inferencecited by 0
  3. 2026Distributional Reinforcement Learning via the Cram'er Distancecited by 0