# RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance

arXiv: https://arxiv.org/abs/2509.25604 · HTML: https://mlsanity.com/paper/2509.25604/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning.
- Builds on 28 of its 67 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- LLM reasoning / [Diffusion reasoning and test-time scaling](https://mlsanity.com/lr/diffusion-reasoning-and-test-time-scaling/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [A General Framework for Inference-time Scaling and Steering of Diffusion Models](https://mlsanity.com/paper/2501.06848/) (2025)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Remasking Discrete Diffusion Models with Inference-Time Scaling](https://mlsanity.com/paper/2503.00307/) (2025)
- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2504.12216/) (2025)
- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding](https://mlsanity.com/paper/2505.16990/) (2025)
- [Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models](https://mlsanity.com/paper/2509.06949/) (2025)
- [LaViDa: A Large Diffusion Language Model for Multimodal Understanding](https://mlsanity.com/paper/2505.16839/) (2025)
- [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](https://mlsanity.com/paper/2507.08838/) (2025)
- [Bayesian Flow Networks](https://mlsanity.com/paper/2308.07037/) (2023)
- [Diffusion Beats Autoregressive in Data-Constrained Settings](https://mlsanity.com/paper/2507.15857/) (2025)
- [Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections](https://mlsanity.com/paper/2507.00018/) (2025)
- [Inference-Time Scaling of Diffusion Language Models via Trajectory Refinement](https://mlsanity.com/paper/2507.08390/) (2025)

## Built on by (most cited first, 7 of 7)

- [Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models](https://mlsanity.com/paper/2602.01842/) (2026)
- [Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones](https://mlsanity.com/paper/2511.15208/) (2025)
- [Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models](https://mlsanity.com/paper/2602.19619/) (2026)
- [Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast](https://mlsanity.com/paper/2605.01373/) (2026)
- [Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs](https://mlsanity.com/paper/2608.00605/) (2026)
- [Unlocking Multimodal Protein Language Models at Inference Time](https://mlsanity.com/paper/2608.25855/) (2026)
- [Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning](https://mlsanity.com/paper/2609.35472/) (2026)

## Abstract (arXiv)

Diffusion Large Language Models (dLLMs) have shown strong reasoning capabilities, yet further improving them typically requires costly post-training with additional data and supervision. We ask whether a post-trained dLLM can improve itself at inference time without additional training, data, or reward models. This requires a guidance signal from the checkpoints alone that is well-defined on the partially masked intermediate states of dLLMs, which existing methods fail to provide. Here we propose Reward-Free Guidance (RFG), a training-free framework for inference-time self-improvement of dLLMs that derives such a signal from the checkpoints themselves. We theoretically demonstrate that a reward signal for a partially masked state can be parameterized by the log-likelihood ratio between a post-trained policy dLLM and its reference model. In practice, this signal can be obtained using off-the-shelf checkpoints alone. Extensive experiments show that RFG consistently improves state-of-the-art post-trained dLLMs by up to 16.1%. Furthermore, despite being entirely training-free, RFG delivers gains that rival or even surpass resource-intensive reinforcement learning techniques.