mlsanityData to 14 Oct 2026

Inpainting-Guided Policy Optimization for Diffusion Large Language Models

arXiv 2509.10396 ↗First posted 12 Sep 2025cs.LGcited by 16

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive LLMs, offering competitive performance while supporting unique generation capabilities such as inpainting. We explore how inpainting can inform RL algorithm design for dLLMs. Aligning LLMs with reinforcement learning faces an exploration challenge: sparse reward signals and sample waste when models fail to discover correct solutions. While this inefficiency affects LLMs broadly, dLLMs offer a distinctive opportunity--their inpainting ability can guide exploration. We introduce IGPO (Inpainting Guided Policy Optimization), an RL framework that strategically inserts partial ground-truth reasoning traces during online sampling. Unlike providing full solutions, inpainting steers exploration toward promising trajectory spaces while preserving self-generated reasoning, bridging supervised fine-tuning and reinforcement learning. We apply IGPO to group-based optimization methods such as GRPO, where exploration failures cause zero advantages and gradients. IGPO restores meaningful gradients while improving sample efficiency. We also propose supervised fine-tuning on synthetically rewritten concise traces that better align with dLLM generation patterns. With additional techniques including entropy-based filtering, our training recipe yields substantial gains across three mathematical benchmarks--GSM8K, Math500, and AMC--achieving new state-of-the-art results for full-attention masked dLLMs.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 37 of its 43 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

37 of 43 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Diffusion Beats Autoregressive in Data-Constrained Settings21 Jul
  2. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  3. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  4. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion27 May
  5. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  7. 2024Scaling up Masked Diffusion Models on Text24 Oct
  8. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  9. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  10. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  11. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
1 more
  1. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  10. 2025Group Sequence Policy Optimizationcited by 835
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  12. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
12 more
  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  2. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  3. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  5. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  6. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  7. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  8. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  9. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  10. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  11. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  12. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  2. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewardscited by 10
  3. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  4. 2025Unifying Deductive and Abductive Reasoning in Knowledge Graphs with Masked Diffusion Modelcited by 8
  5. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewardscited by 7
  6. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  7. 2025Rethinking Reasoning with MDLMs: Early Exits, Post-hoc Reasoning, and Beyondcited by 4
  8. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 4
  9. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  10. 2025C2DLM: Causal Concept-Guided Diffusion Large Language Modelscited by 1
  11. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  12. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
4 more
  1. 2026JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Modelscited by 0
  2. 2026Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Modelscited by 0
  3. 2026CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Modelscited by 0
  4. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
  2. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
  4. 2025Learning Unmasking Policies for Diffusion Language Models
  5. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  6. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
  7. 2025Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models
  8. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executor