mlsanityData to 7 Oct 2026

Simple Policy Gradients for Reasoning with Diffusion Language Models

arXiv 2510.04019 ↗First posted 5 Oct 2025cs.LG cs.AI cs.CLcited by 9

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 1 more
Abstract (arXiv)

Diffusion large language models (dLLMs) represent a promising alternative to autoregressive LLMs; however, the lack of effective post-training techniques, including reinforcement learning (RL), remains a key challenge for dLLMs, especially for downstream applications. Existing approaches often rely on a sequence-level view that requires biased likelihood approximations. In this work, we propose Amortized Group Relative Policy Optimization (AGRPO), a policy gradient algorithm that leverages the Markovian nature of dLLMs, optimizing individual denoising steps rather than full sequences. Our approach improves alignment between the trained policy and the inference process and also admits efficient, unbiased gradient updates via a novel timestep estimation scheme. We demonstrate AGRPO's effectiveness on different math and reasoning tasks, achieving absolute accuracy gains of +59.4% and +69.7% on Countdown and Sudoku over the base LLaDA model, exceeding comparable methods such as diffu-GRPO. Code is available at https://github.com/probablyabot/agrpo.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 27 of its 55 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

27 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul

Other references · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
14 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  6. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  7. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  8. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  9. 2024Diffusion Policy Policy Optimizationcited by 255
  10. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  11. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  12. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  13. 2025Do Large Language Model Benchmarks Test Reliability?cited by 50
  14. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  2. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 2
  3. 2026Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Modelscited by 1
  4. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 0
  5. 2026CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Modelscited by 0
  6. 2026Diffusion Drafts, AR Verifies: Accelerating Document OCR with Self-Speculative Decodingcited by 0
  7. 2026Hesitation-Aware On-Policy Distillation for Diffusion Language Modelscited by 0
  8. 2026Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMscited by 0
  9. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Models
  2. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
  3. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
  4. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
  5. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  6. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
  7. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
  8. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models