mlsanityData to 7 Oct 2026

dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models

arXiv 2603.18806 ↗First posted 19 Mar 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation, which in turn presents new challenges for aligning them with human preferences. In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation, thereby enabling scaled-up offline policy training. We prove that: (i) under reference policy regularization, the probability ratio of the newly unmasked tokens is an unbiased estimate of that of intermediate diffusion states, and (ii) the probability of the full trajectory can be effectively estimated with a single forward pass of a re-masked final state. By integrating these two trajectory reduction strategies into a policy optimization objective, we propose Trajectory Reduction Policy Optimization (dTRPO). We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks. Results show that it substantially improves the core performance of state-of-the-art dLLMs, achieving gains of up to 9.6% on STEM tasks, up to 4.3% on coding tasks, and up to 3.0% on instruction-following tasks. Moreover, dTRPO exhibits strong training efficiency due to its offline, single-forward nature, and achieves improved generation efficiency through high-quality outputs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 58 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

50 of 58 references.

References · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
38 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2025Large Language Diffusion Modelscited by 872
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  10. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  13. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  14. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  15. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  16. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  17. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  18. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  19. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  20. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  21. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  22. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  23. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
  24. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  25. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 80
  26. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  27. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  28. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  29. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 38
  30. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  31. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  32. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 32
  33. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  34. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  35. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 22
  36. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 17
  37. 2025Thinking Inside the Mask: In-Place Prompting in Diffusion LLMscited by 13
  38. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Modelscited by 1
  2. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  3. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  2. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
  4. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  5. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executor
  6. 2025Learning Unmasking Policies for Diffusion Language Models
  7. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Models
  8. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models