mlsanityData to 14 Oct 2026

Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models

arXiv 2610.00661 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Diffusion reasoning and test-time scaling also fits: Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback across these decisions, even as their conditioning context changes. We propose stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer. For binary rewards, we show that this transformation is exactly a prompt- and state-dependent rescaling of centered outcome advantages. A capability-based calibration suggests a coefficient scale, while endpoint and interpolation ablations guide schedule selection. Across multiple dLLM backbones and mathematical reasoning benchmarks, StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity. In our ablation studies, mass-matched controls support the contributions of state allocation and schedule direction, and the gains persist after matching the root mean square (RMS) of the advantages to that of centered GRPO. Reasoning-trace diagnostics further show that the diversity gains from StepRS-GRPO extend beyond final-answer strings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 33 of its 53 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

33 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models21 Jan
  2. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization9 Oct
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization9 Oct
  4. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models5 Oct
  5. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  6. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2025Large Language Diffusion Modelscited by 912
  7. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  9. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  10. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  11. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 88
  12. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
15 more
  1. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  2. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  3. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  4. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  5. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  6. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 14
  7. 2026dLLM: Simple Diffusion Language Modelingcited by 14
  8. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Trainingcited by 12
  9. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Trainingcited by 12
  10. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradientscited by 11
  11. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  12. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 10
  13. 2025Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Modelscited by 9
  14. 2026Diffusion-State Policy Optimization for Masked Diffusion Language Modelscited by 4
  15. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
  2. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
  3. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
  4. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
  5. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  6. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  7. 2025MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization
  8. 2026Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning