mlsanityData to 7 Oct 2026

d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

arXiv 2504.12216 ↗First posted 16 Apr 2025cs.CL cs.LGcited by 178

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

Abstract (arXiv)

Recent large language models (LLMs) have demonstrated strong reasoning capabilities that benefits from online reinforcement learning (RL). These capabilities have primarily been demonstrated within the left-to-right autoregressive (AR) generation paradigm. In contrast, non-autoregressive paradigms based on diffusion generate text in a coarse-to-fine manner. Although recent diffusion-based large language models (dLLMs) have achieved competitive language modeling performance compared to their AR counterparts, it remains unclear if dLLMs can also leverage recent advances in LLM reasoning. To this end, we propose d1, a framework to adapt pre-trained masked dLLMs into reasoning models via a combination of supervised finetuning (SFT) and RL. Specifically, we develop and extend techniques to improve reasoning in pretrained dLLMs: (a) we utilize a masked SFT technique to distill knowledge and instill self-improvement behavior directly from existing datasets, and (b) we introduce a novel critic-free, policy-gradient based RL algorithm called diffu-GRPO, the first integration of policy gradient methods to masked dLLMs. Through empirical studies, we investigate the performance of different post-training recipes on multiple mathematical and planning benchmarks. We find that d1 yields the best performance and significantly improves performance of a state-of-the-art dLLM. Our code is released at https://dllm-reasoning.github.io/.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 37 of its 49 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

37 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  2. 2024Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models12 Feb

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Let's Verify Step by Stepcited by 3,598
23 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  4. 2025s1: Simple test-time scalingcited by 1,311
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  7. 2025Large Language Diffusion Modelscited by 872
  8. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  11. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  12. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  13. 2025LIMO: Less is More for Reasoningcited by 488
  14. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  15. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  16. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  17. 2024Scaling up Masked Diffusion Models on Textcited by 195
  18. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  19. 2023Likelihood-Based Diffusion Language Modelscited by 147
  20. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  21. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  22. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 36
  23. 2025Dynamic Scaling of Unit Tests for Code Reward Modelingcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 73; 2026: 105.

178 papers.

Sort:

By area:

  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  2. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  3. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  7. 2025A Survey on Diffusion Language Modelscited by 75
  8. 2025A Survey on Latent Reasoningcited by 71
  9. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  10. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  11. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  12. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
38 more
  1. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  2. 2025Any-Order Flexible Length Masked Diffusioncited by 48
  3. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 45
  4. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  5. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  6. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  7. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  8. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  9. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Computecited by 36
  10. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  11. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  12. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32
  13. 2025Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationcited by 30
  14. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 30
  15. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  16. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  17. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  18. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  19. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  20. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  21. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 21
  22. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 21
  23. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  24. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  25. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 17
  26. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  27. 2025Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Stepcited by 16
  28. 2025DINGO: Constrained Inference for Diffusion LLMscited by 15
  29. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 15
  30. 2025Soft-Masked Diffusion Language Modelscited by 15
  31. 2025Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Modelscited by 15
  32. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 14
  33. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 14
  34. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  35. 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
  36. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  37. 2026dLLM: Simple Diffusion Language Modelingcited by 12
  38. 2026Continuous Latent Diffusion Language Modelcited by 12

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
  2. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  3. 2026Trace-Based On-Policy Distillation for Masked Diffusion Language Models
  4. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  5. 2025Training Language Models to Reason Efficiently
  6. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  7. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models
  8. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning