# Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

arXiv: https://arxiv.org/abs/2509.06949 · HTML: https://mlsanity.com/paper/2509.06949/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- First posted on arXiv 8 Sep 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Built on by (most cited first, 30 of 88)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [LLaDA2.0: Scaling Up Diffusion Language Models to 100B](https://mlsanity.com/paper/2512.15745/) (2025)
- [dParallel: Learnable Parallel Decoding for dLLMs](https://mlsanity.com/paper/2509.26488/) (2025)
- [LLaDA-MoE: A Sparse MoE Diffusion Language Model](https://mlsanity.com/paper/2509.24389/) (2025)
- [LLaDA2.1: Speeding Up Text Diffusion via Token Editing](https://mlsanity.com/paper/2602.08676/) (2026)
- [SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models](https://mlsanity.com/paper/2510.09541/) (2025)
- [d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation](https://mlsanity.com/paper/2601.07568/) (2026)
- [Demystifying Reinforcement Learning in Agentic Reasoning](https://mlsanity.com/paper/2510.11701/) (2025)
- [Learning Unmasking Policies for Diffusion Language Models](https://mlsanity.com/paper/2512.09106/) (2025)
- [d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation](https://mlsanity.com/paper/2509.21474/) (2025)
- [RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System](https://mlsanity.com/paper/2602.02488/) (2026)
- [Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs](https://mlsanity.com/paper/2509.18085/) (2025)
- [Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization](https://mlsanity.com/paper/2510.08233/) (2025)
- [Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models](https://mlsanity.com/paper/2510.00294/) (2025)
- [DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning](https://mlsanity.com/paper/2510.02212/) (2025)
- [DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving](https://mlsanity.com/paper/2602.14577/) (2026)
- [d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models](https://mlsanity.com/paper/2512.09675/) (2025)
- [dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning](https://mlsanity.com/paper/2512.04459/) (2025)
- [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://mlsanity.com/paper/2606.06712/) (2026)
- [Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards](https://mlsanity.com/paper/2510.01544/) (2025)
- [dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learning](https://mlsanity.com/paper/2512.21446/) (2025)
- [DiRL: An Efficient Post-Training Framework for Diffusion Language Models](https://mlsanity.com/paper/2512.22234/) (2025)
- [Residual Context Diffusion Language Models](https://mlsanity.com/paper/2601.22954/) (2026)
- [Stabilizing Reinforcement Learning for Diffusion Language Models](https://mlsanity.com/paper/2603.06743/) (2026)
- [MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation](https://mlsanity.com/paper/2511.09611/) (2025)
- [MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding](https://mlsanity.com/paper/2603.22458/) (2026)
- [RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance](https://mlsanity.com/paper/2509.25604/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models](https://mlsanity.com/paper/2602.14147/) (2026)
- [Improving Sampling for Masked Diffusion Models via Information Gain](https://mlsanity.com/paper/2602.18176/) (2026)

## Abstract (arXiv)

We propose TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that incorporates preferred inference trajectory into post-training, and is applicable across different architectures. Equipped with a diffusion-based value model that enhances training stability, we demonstrate improved reasoning performance on complex math and coding tasks. Besides, it can also be applied to adapt block-specific models to larger blocks, which improves sampling flexibility. Employing TraceRL, we derive a series of state-of-the-art diffusion language models, namely TraDo. Although smaller than 7B-scale AR models, TraDo-4B-Instruct still consistently outperforms them across complex math reasoning tasks. TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical reasoning benchmarks. Through curriculum learning, we also derive the first long-CoT DLM, outperforming Qwen2.5-7B-Instruct on MATH500 with an 18.1% relative accuracy gain. To facilitate reproducible research and practical applications, we release a comprehensive open-source framework for building, training, and deploying diffusion LLMs across diverse architectures. The framework integrates accelerated KV-cache techniques and inference engines for both inference and reinforcement learning, and includes implementations of various supervised fine-tuning and RL methods for mathematics, coding, and general tasks. Code and Models: https://github.com/Gen-Verse/dLLM-RL