mlsanityData to 7 Oct 2026

Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology

arXiv 2509.04372 ↗First posted 4 Sep 2025stat.ML cs.GL cs.LGcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

In this note, we reflect on several fundamental connections among widely used post-training techniques. We clarify some intimate connections and equivalences between reinforcement learning with human feedback, reinforcement learning with internal feedback, and test-time scaling (particularly soft best-of-N sampling), while also illuminating intrinsic links between diffusion guidance and test-time scaling. Additionally, we introduce a resampling approach for alignment and reward-directed diffusion models, sidestepping the need for explicit reinforcement learning techniques.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 23 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

18 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Learning to Reason without External Rewards26 May

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2025s1: Simple test-time scalingcited by 1,311
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  10. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  11. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  12. 2020Spectral Methods for Data Science: A Statistical Perspectivecited by 111
5 more
  1. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  2. 2024Classifier-Free Guidance is a Predictor-Correctorcited by 79
  3. 2025Soft Best-of-n Sampling for Model Alignmentcited by 28
  4. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  5. 2024A Score-Based Density Formula, with Applications in Diffusion Generative Modelscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Towards a unified framework for guided diffusion modelscited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Models for Reinforcement Learning: A Survey
  2. 2026Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
  3. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  4. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
  5. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
  6. 2025Policy Gradient Guidance Enables Test Time Control
  7. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
  8. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matching