mlsanityData to 14 Oct 2026

D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks

arXiv 2510.17212 ↗First posted 20 Oct 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 24 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

19 of 24 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  9. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  10. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  11. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  12. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
7 more
  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  2. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  3. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
  4. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 44
  5. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 38
  6. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 36
  7. 2020FORK: A Forward-Looking Actor For Model-Free Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning
  2. 2025Categorical Policies: Multimodal Policy Learning and Exploration in Continuous Control
  3. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  4. 2026Dual-Flow Reinforcement Learning with State-Aware Exploration
  5. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  6. 2026FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion
  7. 2024Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation
  8. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?