mlsanityData to 7 Oct 2026

Reinforcement Learning from Denoising Feedback

arXiv 2605.25638 ↗First posted 25 May 2026cs.CL cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel training paradigm that leverages feedback obtained from rollout and training processes to facilitate accurate and efficient policy loss estimation. To balance the trade-off between computational efficiency and estimation effectiveness, RLDF optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that RLDF achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work lays a principled foundation for scalable reinforcement learning in diffusion language models. We build Drift, a training framework for DLMs, available at https://github.com/ant-research/Drift.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 49 of its 59 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

49 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models25 May

Other references · 48

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
36 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2025Large Language Diffusion Modelscited by 872
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  8. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  9. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  10. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  11. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  13. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  14. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  15. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  16. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
  17. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  18. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  19. 2022Continuous diffusion for categorical datacited by 202
  20. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  21. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  22. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  23. 2023Likelihood-Based Diffusion Language Modelscited by 147
  24. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  25. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  26. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  27. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  28. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 80
  29. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  30. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  31. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  32. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  33. 2025Diffusion Language Models are Super Data Learnerscited by 45
  34. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  35. 2025What Makes Diffusion Language Models Super Data Learners?cited by 8
  36. 2025Veri-R1: Toward Precise and Faithful Claim Verification via Online Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
  2. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
  3. 2026Read the Trace, Steer the Path: Trajectory-Aware Reinforcement Learning for Diffusion Language Models
  4. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
  5. 2026DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
  6. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
  7. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Process
  8. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models