mlsanityData to 14 Oct 2026

RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance

arXiv 2509.25604 ↗First posted 29 Sep 2025cs.CL cs.LGcited by 7

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 1 more
Abstract (arXiv)

Diffusion Large Language Models (dLLMs) have shown strong reasoning capabilities, yet further improving them typically requires costly post-training with additional data and supervision. We ask whether a post-trained dLLM can improve itself at inference time without additional training, data, or reward models. This requires a guidance signal from the checkpoints alone that is well-defined on the partially masked intermediate states of dLLMs, which existing methods fail to provide. Here we propose Reward-Free Guidance (RFG), a training-free framework for inference-time self-improvement of dLLMs that derives such a signal from the checkpoints themselves. We theoretically demonstrate that a reward signal for a partially masked state can be parameterized by the log-likelihood ratio between a post-trained policy dLLM and its reference model. In practice, this signal can be obtained using off-the-shelf checkpoints alone. Extensive experiments show that RFG consistently improves state-of-the-art post-trained dLLMs by up to 16.1%. Furthermore, despite being entirely training-free, RFG delivers gains that rival or even surpass resource-intensive reinforcement learning techniques.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 28 of its 67 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

28 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Inference-Time Scaling of Diffusion Language Models via Trajectory Refinement11 Jul
  2. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  3. 2025Mercury: Ultra-Fast Language Models Based on Diffusion17 Jun
  4. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr
  5. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  6. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Models12 Jan

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  8. 2025Large Language Diffusion Modelscited by 912
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  10. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
10 more
  1. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  2. 2022Continuous diffusion for categorical datacited by 214
  3. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  5. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  6. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 88
  7. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  8. 2023Bayesian Flow Networkscited by 63
  9. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 63
  10. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Modelscited by 7
  2. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  3. 2026Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Modelscited by 1
  4. 2026Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrastcited by 0
  5. 2026Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMscited by 0
  6. 2026Unlocking Multimodal Protein Language Models at Inference Timecited by 0
  7. 2026Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
  2. 2026Relative Score Policy Optimization for Diffusion Language Models
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
  4. 2025MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization
  5. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  6. 2026Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning
  7. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
  8. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation