mlsanityData to 14 Oct 2026

RL Post-Training Diffusion Language Models Along Their Own Generation Paths

arXiv 2610.17266 ↗First posted 12 Oct 2026cs.AIcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion language models are a promising and more efficient alternative to autoregressive models. However, most RL post-training techniques are not easily adapted to these models. One issue is that diffusion language models generate answers through many intermediate states, deciding at each step which masked positions to reveal and which tokens to place in them. Many existing RL methods used in this setting approximate intractable sequence likelihoods or weight every denoising step by the same advantage, without explicitly distinguishing how these intermediate states of the denoising trajectory affect the expected reward. To address this issue of trajectory blindness, we introduce PACT (Potential-Anchored Consistency of Transitions). Motivated by a reward-tilted reference distribution, PACT jointly learns the policy and a scalar potential that scores each intermediate state, anchored by terminal reward. For each sampled denoising transition, a consistency loss matches the policy-to-reference log-probability ratio of revealed tokens to the change in this potential. This formulation requires neither sequence-likelihood estimation nor importance-ratio clipping. On four tasks, PACT improves LLaDA-8B-Instruct by +9.9 points on GSM8K, +12.0 on MATH500, +55.7 on Countdown, and +72.6 on Sudoku, and reaches the best average accuracy across the four tasks reported for this model, +3.7 points over the next best method. Further compared to AGRPO, it reaches that baseline's best accuracy in about half the training compute on GSM8K.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 33 of its 36 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

33 of 36 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale9 Sep
  2. 2026Mask-Aware Policy Gradients for Diffusion Language Models16 Jul
  3. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models28 May
  4. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models13 May
  5. 2026Relative Score Policy Optimization for Diffusion Language Models11 May
  6. 2026Discrete Tilt Matching20 Apr
  7. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone19 Nov
  8. 2025On the Reasoning Abilities of Masked Diffusion Language Models15 Oct
  9. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models10 Oct
  10. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models5 Oct
  11. 2025Dream 7B: Diffusion Large Language Models21 Aug
  12. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
2 more
  1. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
  2. 2024Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models12 Feb

Other references · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  6. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  7. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  8. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  9. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  10. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  11. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  12. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
7 more
  1. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  2. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 32
  3. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  4. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 30
  5. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  6. 2025Relative Trajectory Balance is equivalent to Trust-PCLcited by 5
  7. 2026GFlowRL: Scaling Distribution-Matching RL to Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
  2. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
  3. 2026Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models
  4. 2026On Trajectory-Aware Training for Masked Diffusion Language Models
  5. 2026Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  7. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  8. 2026Commit Locally, Exit Globally: Coordinating Adaptive Sampling and Early Exit in Diffusion Language Models