mlsanityData to 14 Oct 2026

Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces

arXiv 2509.22963 ↗First posted 26 Sep 2025cs.LGcited by 7

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) struggles to scale to large, combinatorial action spaces common in many real-world problems. This paper introduces a novel framework for training discrete diffusion models as highly effective policies in these complex settings. Our key innovation is an efficient online training process that ensures stable and effective policy improvement. By leveraging policy mirror descent (PMD) to define an ideal, regularized target policy distribution, we frame the policy update as a distributional matching problem, training the expressive diffusion model to replicate this stable target. This decoupled approach stabilizes learning and significantly enhances training performance. Our method achieves state-of-the-art results and superior sample efficiency across a diverse set of challenging combinatorial benchmarks, including DNA sequence generation, RL with macro-actions, and multi-agent systems. Experiments demonstrate that our diffusion policies attain superior performance compared to other baselines.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 71 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

53 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning4 Feb
  2. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods3 Feb
  3. 2024Diffusion Policy Policy Optimization1 Sep
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  5. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  11. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  12. 2025Large Language Diffusion Modelscited by 912
35 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  5. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  6. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  8. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  9. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  12. 2018Maximum a Posteriori Policy Optimisationcited by 328
  13. 2018Distributed Prioritized Experience Replaycited by 269
  14. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  15. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 232
  16. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  17. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 177
  18. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 169
  19. 2023Protein Design with Guided Discrete Diffusioncited by 157
  20. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  21. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  22. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 150
  23. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  24. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  25. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  26. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  27. 2020Mirror Descent Policy Optimizationcited by 106
  28. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  29. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  30. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  31. 2019The Natural Language of Actionscited by 28
  32. 2013Hierarchical Solution of Markov Decision Processes using Macro-actionscited by 19
  33. 2016Deep Reinforcement Learning With Macro-Actionscited by 14
  34. 2023Representation-Driven Reinforcement Learningcited by 4
  35. 2023Boosting Studies of Multi-Agent Reinforcement Learning on Google Research Football Environment: the Past, Present, and Futurecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  2. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 3
  3. 2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RLcited by 2
  4. 2026Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actionscited by 1
  5. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  6. 2026Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradientscited by 1
  7. 2026From Noise to Control: Parameterized Diffusion Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
  2. 2025One-Step Flow Policy Mirror Descent
  3. 2026CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space
  4. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  5. 2025Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data
  6. 2025Efficient Online Reinforcement Learning for Diffusion Policy
  7. 2026Approximation of Log-Partition Function in Policy Mirror Descent Induces Implicit Regularization for LLM Post-Training
  8. 2026CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance