mlsanityData to 7 Oct 2026

Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

arXiv 2603.12554 ↗First posted 13 Mar 2026cs.LG cs.AI cs.CLcited by 3

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Reinforcement learning (RL) has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods. Existing approaches therefore rely on surrogate likelihoods or heuristic approximations, which can introduce bias and obscure the sequential structure of denoising. We formulate diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive an exact, unbiased policy gradient that decomposes over denoising steps and is expressed in terms of intermediate advantages, without requiring explicit evaluation of the sequence likelihood. To obtain a practical and compute-efficient estimator, we (i) select denoising steps for policy updates via an entropy-guided approximation bound, and (ii) estimate intermediate advantages using a one-step denoising reward naturally provided by the diffusion model, avoiding costly multi-step rollouts. Experiments on coding and logical reasoning benchmarks demonstrate state-of-the-art results, with strong competitive performance on mathematical reasoning, outperforming existing RL post-training approaches for DLMs. Code is available at https://github.com/vishnutez/egspo-dllm-rl.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 35 of its 38 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

35 of 38 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models10 Oct
  2. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies7 Oct
  3. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning2 Oct
  4. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation25 Sep
  5. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models8 Sep
  6. 2025Dream 7B: Diffusion Large Language Models21 Aug
  7. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  8. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  9. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs17 Jun
  10. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking30 May
  11. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  12. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
5 more
  1. 2025Large Language Diffusion Models14 Feb
  2. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  3. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  10. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  12. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
6 more
  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  2. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  5. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  6. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learnability-Informed Fine-Tuning of Diffusion Language Modelscited by 3
  2. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Modelscited by 2
  3. 2026Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
  2. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
  3. 2026Reinforcement Learning from Denoising Feedback
  4. 2025Learning Unmasking Policies for Diffusion Language Models
  5. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
  6. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
  7. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  8. 2026DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models