mlsanityData to 14 Oct 2026

d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation

arXiv 2509.21474 ↗First posted 25 Sep 2025cs.LGcited by 27

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 1 more
Abstract (arXiv)

While diffusion language models (DLMs) have achieved competitive performance in text generation, improving their reasoning ability with reinforcement learning remains an active research area. Here, we introduce d2, a reasoning framework tailored for masked DLMs. Central to our framework is a new policy gradient algorithm that relies on accurate estimates of the sampling trajectory likelihoods. Because computing these likelihoods naively is computationally expensive for masked DLMs, we develop a family of estimators tailored to distinct model classes. For DLMs that support a sampling algorithm called any-order decoding, we propose d2-AnyOrder, which achieves exact trajectory likelihood with a single model pass. Through an empirical study of widely used DLMs, we show that any-order decoding is not universally supported in practice. For standard masked diffusion models, we propose d2-StepMerge, which approximates the trajectory likelihood, trading off compute for approximation accuracy in an analytically tractable manner. Empirically, d2 significantly outperforms widely-used RL baselines when applied to popular DLMs, and sets a new state-of-the-art performance for DLMs on logical reasoning tasks (Countdown and Sudoku) and math reasoning benchmarks (GSM8K and MATH500). We provide the code along with a blog post on the project page: https://guanghanwang.com/d2

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 38 of its 56 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

38 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  2. 2025Mercury: Ultra-Fast Language Models Based on Diffusion17 Jun
  3. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr
  4. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  5. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Models12 Jan

Other references · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
21 more
  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  8. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  9. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  12. 2021Autoregressive Diffusion Modelscited by 210
  13. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  14. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  15. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  16. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  17. 2025The Diffusion Dualitycited by 135
  18. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  19. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 88
  20. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  21. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 22.

27 papers.

Sort:

By area:

  1. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  2. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 24
  3. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 17
  4. 2026DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmaskingcited by 9
  5. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  6. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  7. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  8. 2026A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Modelscited by 5
  9. 2026Diffusion-State Policy Optimization for Masked Diffusion Language Modelscited by 4
  10. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantagescited by 4
  11. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  12. 2025Dimension-free error estimate for diffusion model and optimal schedulingcited by 3
15 more
  1. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  2. 2026ReflectDrive-2: Reinforcement-Learning-Aligned Self-Editing for Discrete Diffusion Drivingcited by 3
  3. 2026Learnability-Informed Fine-Tuning of Diffusion Language Modelscited by 3
  4. 2026Discrete Tilt Matchingcited by 2
  5. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusioncited by 2
  6. 2026VDLM: Variable Diffusion LMs via Robust Latent-to-Text Renderingcited by 1
  7. 2026Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learningcited by 1
  8. 2026The Confidence Shortcut: A Reasoning Failure Mode of Masked Diffusion Modelscited by 1
  9. 2026Hesitation-Aware On-Policy Distillation for Diffusion Language Modelscited by 1
  10. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 0
  11. 2026Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Modelscited by 0
  12. 2026SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicingcited by 0
  13. 2026CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Modelscited by 0
  14. 2026Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMscited by 0
  15. 2026Conditional Generation of Creative Chess Puzzles with Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mask-Aware Policy Gradients for Diffusion Language Models
  2. 2025Diffusion Language Model Inference with Monte Carlo Tree Search
  3. 2026Diffusion Language Models: An Experimental Analysis
  4. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
  5. 2026The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
  6. 2026Hierarchical Continuous Diffusion Language Models
  7. 2025A Survey on Diffusion Language Models
  8. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques