mlsanityData to 13 Oct 2026

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

arXiv 2509.06949 ↗First posted 8 Sep 2025cs.CLcited by 84

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We propose TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that incorporates preferred inference trajectory into post-training, and is applicable across different architectures. Equipped with a diffusion-based value model that enhances training stability, we demonstrate improved reasoning performance on complex math and coding tasks. Besides, it can also be applied to adapt block-specific models to larger blocks, which improves sampling flexibility. Employing TraceRL, we derive a series of state-of-the-art diffusion language models, namely TraDo. Although smaller than 7B-scale AR models, TraDo-4B-Instruct still consistently outperforms them across complex math reasoning tasks. TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical reasoning benchmarks. Through curriculum learning, we also derive the first long-CoT DLM, outperforming Qwen2.5-7B-Instruct on MATH500 with an 18.1% relative accuracy gain. To facilitate reproducible research and practical applications, we release a comprehensive open-source framework for building, training, and deploying diffusion LLMs across diverse architectures. The framework integrates accelerated KV-cache techniques and inference engines for both inference and reinforcement learning, and includes implementations of various supervised fine-tuning and RL methods for mathematics, coding, and general tasks. Code and Models: https://github.com/Gen-Verse/dLLM-RL

Key facts. Cited by 84 papers in the map (data through 13 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. First posted on arXiv 8 Sep 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 23; 2026: 61.

84 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 177
  2. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 161
  3. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 60
  4. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 56
  5. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 51
  6. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 40
  7. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 35
  8. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  9. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 30
  10. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  11. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  12. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 20
38 more
  1. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Modelscited by 19
  2. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  3. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 17
  4. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 17
  5. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 16
  6. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  7. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewardscited by 10
  8. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10
  9. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  10. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 10
  11. 2026Residual Context Diffusion Language Modelscited by 9
  12. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 9
  13. 2025MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generationcited by 8
  14. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidancecited by 7
  15. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  16. 2026Improving Sampling for Masked Diffusion Models via Information Gaincited by 7
  17. 2026MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decodingcited by 7
  18. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
  19. 2026LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learningcited by 6
  20. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  21. 2026A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Modelscited by 5
  22. 2026Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Drivingcited by 5
  23. 2026DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Modelscited by 5
  24. 2025CoDA: Coding LM via Diffusion Adaptationcited by 4
  25. 2025Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Modelscited by 4
  26. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 4
  27. 2026Learning from the Self-future: On-policy Self-distillation for dLLMscited by 4
  28. 2026AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafterscited by 4
  29. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  30. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  31. 2026FlashBlock: Attention Caching for Efficient Long-Context Block Diffusioncited by 3
  32. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 3
  33. 2026Few-Step Diffusion Language Models via Trajectory Self-Distillationcited by 3
  34. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantagescited by 3
  35. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  36. 2026Training-Free Self-Correction for Multimodal Masked Diffusion Modelscited by 2
  37. 2026Adaptive Test-Time Compute Allocation for Block Diffusion Language Models in Complex Reasoningcited by 2
  38. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Modelscited by 2

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
  2. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
  3. 2026T^star: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning
  4. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models
  5. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
  6. 2026Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching
  7. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  8. 2026Breaking the Factorization Barrier in Diffusion Language Models