# Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models

arXiv: https://arxiv.org/abs/2610.00661 · HTML: https://mlsanity.com/paper/2610.00661/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning.
- Builds on 33 of its 53 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM reasoning / [Diffusion reasoning and test-time scaling](https://mlsanity.com/lr/diffusion-reasoning-and-test-time-scaling/)
- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2504.12216/) (2025)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation](https://mlsanity.com/paper/2510.06303/) (2025)
- [Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models](https://mlsanity.com/paper/2509.06949/) (2025)
- [Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems](https://mlsanity.com/paper/2505.15201/) (2025)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](https://mlsanity.com/paper/2507.08838/) (2025)
- [SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models](https://mlsanity.com/paper/2510.09541/) (2025)
- [The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward](https://mlsanity.com/paper/2509.07430/) (2025)
- [Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective](https://mlsanity.com/paper/2512.03759/) (2025)
- [Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization](https://mlsanity.com/paper/2510.08554/) (2025)
- [Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization](https://mlsanity.com/paper/2510.08233/) (2025)
- [The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models](https://mlsanity.com/paper/2601.15165/) (2026)
- [Inpainting-Guided Policy Optimization for Diffusion Large Language Models](https://mlsanity.com/paper/2509.10396/) (2025)
- [Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models](https://mlsanity.com/paper/2509.24261/) (2025)
- [dLLM: Simple Diffusion Language Modeling](https://mlsanity.com/paper/2602.22661/) (2026)
- [RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training](https://mlsanity.com/paper/2510.00911/) (2025)
- [Simple Policy Gradients for Reasoning with Diffusion Language Models](https://mlsanity.com/paper/2510.04019/) (2025)
- [SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training](https://mlsanity.com/paper/2510.08141/) (2025)
- [Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients](https://mlsanity.com/paper/2510.23049/) (2025)
- [DiRL: An Efficient Post-Training Framework for Diffusion Language Models](https://mlsanity.com/paper/2512.22234/) (2025)
- [Stabilizing Reinforcement Learning for Diffusion Language Models](https://mlsanity.com/paper/2603.06743/) (2026)

## Abstract (arXiv)

Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback across these decisions, even as their conditioning context changes. We propose stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer. For binary rewards, we show that this transformation is exactly a prompt- and state-dependent rescaling of centered outcome advantages. A capability-based calibration suggests a coefficient scale, while endpoint and interpolation ablations guide schedule selection. Across multiple dLLM backbones and mathematical reasoning benchmarks, StepRS-GRPO improves both pass@1 accuracy and pass@k coverage over centered GRPO, while increasing answer diversity. In our ablation studies, mass-matched controls support the contributions of state allocation and schedule direction, and the gains persist after matching the root mean square (RMS) of the advantages to that of centered GRPO. Reasoning-trace diagnostics further show that the diversity gains from StepRS-GRPO extend beyond final-answer strings.