mlsanityData to 7 Oct 2026

d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models

arXiv 2512.09675 ↗First posted 10 Dec 2025cs.CLcited by 14

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) is pivotal for enhancing the reasoning capabilities of diffusion large language models (dLLMs). However, existing dLLM policy optimization methods suffer from two critical reliability bottlenecks: (1) reward sparsity, arising from coarse or unverifiable signals that impede accurate advantage calculation; and (2) their probability estimates do not account for the gap to the unbiased expectation over all decoding orders, which are intractable to compute. To mitigate these issues, we propose d-TreeRPO, a reliable RL framework for dLLMs that leverages tree-structured rollouts and bottom-up advantage computation based on verifiable outcome rewards to provide fine-grained and verifiable step-wise reward signals. Furthermore, we provide a theoretical proof demonstrating that increasing prediction confidence effectively minimizes the gap between unbiased expected prediction probabilities and its single-step forward pass estimate. Guided by this analysis, we introduce a time-scheduled self-distillation loss during training that enhances prediction confidence in later training stages, thereby enabling more accurate probability estimation and better performance. Experiments demonstrate that d-TreeRPO outperforms existing baselines and achieves significant improvements across multiple reasoning benchmarks. Specifically, it achieves +86.2% on Sudoku, +51.6% on Countdown, +4.5% on GSM8K, and +5.3% on Math500 compared to the base model.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 30 of its 49 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

30 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization9 Oct
  2. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards2 Oct
  3. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation25 Sep
  4. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  5. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
  6. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2025Large Language Diffusion Modelscited by 872
  7. 2025Group Sequence Policy Optimizationcited by 788
  8. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  9. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
12 more
  1. 2024Scaling up Masked Diffusion Models on Textcited by 195
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  4. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  5. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  6. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 80
  7. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  8. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  9. 2024Entropy-Regularized Process Reward Modelcited by 38
  10. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  11. 2025Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Stepcited by 16
  12. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  2. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  3. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 14
  4. 2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Modelscited by 5
  5. 2026From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discoverycited by 4
  6. 2026Learning from the Self-future: On-policy Self-distillation for dLLMscited by 4
  7. 2026Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuningcited by 2
  8. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 2
  9. 2026TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decodingcited by 2
  10. 2026Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Modelscited by 1
  11. 2026Relative Score Policy Optimization for Diffusion Language Modelscited by 0
  12. 2026Towards In-Depth Root Cause Localization for Microservices with Multi-Agent Recursion-of-Thoughtcited by 0
2 more
  1. 2026ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillationcited by 0
  2. 2026Hesitation-Aware On-Policy Distillation for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
  2. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
  3. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
  4. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  5. 2025Learning Unmasking Policies for Diffusion Language Models
  6. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
  7. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
  8. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executor