mlsanityData to 7 Oct 2026

Distributions as Actions: A Unified Framework for Diverse Action Spaces

arXiv 2506.16608 ↗First posted 19 Jun 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce a novel reinforcement learning (RL) framework that treats parameterized action distributions as actions, redefining the boundary between agent and environment. This reparameterization makes the new action space continuous, regardless of the original action type (discrete, continuous, hybrid, etc.). Under this new parameterization, we develop a generalized deterministic policy gradient estimator, Distributions-as-Actions Policy Gradient (DA-PG), which has lower variance than the gradient in the original action space. Although learning the critic over distribution parameters poses new challenges, we introduce Interpolated Critic Learning (ICL), a simple yet effective strategy to enhance learning, supported by insights from bandit settings. Building on TD3, a strong baseline for continuous control, we propose a practical actor-critic algorithm, Distributions-as-Actions Actor-Critic (DA-AC). Empirically, DA-AC achieves competitive performance in various settings across discrete, continuous, and hybrid control.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

42 of 63 references.

References · 42

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
30 more
  1. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  3. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  4. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  5. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  6. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  7. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 65
  8. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  9. 2015Reinforcement Learning with Parameterized Actionscited by 61
  10. 2018Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Spacecited by 54
  11. 2023Empirical Design in Reinforcement Learningcited by 54
  12. 2019Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Spacecited by 53
  13. 2018Variance reduction properties of the reparameterization trickcited by 41
  14. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  15. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  16. 2022Solving Continuous Control via Q-learningcited by 21
  17. 2019On Value Functions and the Agent-Environment Boundarycited by 19
  18. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  19. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  20. 2019Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spacescited by 15
  21. 2017Expected Policy Gradientscited by 14
  22. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  23. 2017Mean Actor Criticcited by 8
  24. 2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
  25. 2025Agency Is Frame-Dependentcited by 5
  26. 2023Representation-Driven Reinforcement Learningcited by 4
  27. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
  28. 2024q-exponential family for policy optimizationcited by 2
  29. 2025Equivalence of stochastic and deterministic policy gradientscited by 2
  30. 2025Dynamical System Optimizationcited by 1

Built on by

Not cited yet.