mlsanityData to 14 Oct 2026

FASTER: Value-Guided Sampling for Fast RL

arXiv 2604.19730 ↗First posted 21 Apr 2026cs.LG cs.AIcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Some of the most performant reinforcement learning algorithms today can be prohibitively expensive as they use test-time scaling methods such as sampling multiple action candidates and selecting the best one. In this work, we propose FASTER, a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost by tracing the performance gain of action samples back to earlier in the denoising process. Our key insight is that we can model the denoising of multiple action candidates and selecting the best one as a Markov Decision Process (MDP) where the goal is to progressively filter action candidates before denoising is complete. With this MDP, we can learn a policy and value function in the denoising space that predicts the downstream value of action candidates in the denoising process and filters them while maximizing returns. The result is a method that is lightweight and can be plugged into existing generative RL algorithms. Across challenging long-horizon manipulation tasks in online and batch-online RL, FASTER consistently improves the underlying policies and achieves the best overall performance among the compared methods. Applied to a pretrained VLA, FASTER achieves the same performance while substantially reducing training and inference compute requirements. Code is available at https://github.com/alexanderswerdlow/faster .

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 38 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

36 of 38 references.

References · 36

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  7. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  9. 2024One Step Diffusion via Shortcut Modelscited by 391
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  12. 2025π*0.6: a VLA That Learns From Experiencecited by 303
24 more
  1. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  2. 2025Flow Q-Learningcited by 184
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  4. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 142
  5. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  6. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  7. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  8. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 106
  9. 2024Golden Noise for Diffusion Models: A Learning Frameworkcited by 92
  10. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 89
  11. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  12. 2024Not All Noises Are Created Equally:Diffusion Noise Selection and Optimizationcited by 71
  13. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 61
  14. 2024A Noise is Worth Diffusion Guidancecited by 49
  15. 2026Q-learning with Adjoint Matchingcited by 37
  16. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  17. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  18. 2024FIND: Fine-tuning Initial Noise Distribution with Policy Optimization for Diffusion Modelscited by 20
  19. 2024The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noisecited by 19
  20. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 15
  21. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  22. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  23. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  24. 2025Noise-Level Diffusion Guidance: Well Begun is Half Donecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 11
  2. 2026ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policiescited by 3
  3. 2026Q-Learning With World Modelscited by 2
  4. 2026Reinforcement Learning for Real-Time Vision-Language-Action Policiescited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Denoising Feedback
  2. 2026dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
  3. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps
  4. 2026Learning When to Stop: Prefix-Optimal Dynamic Diffusion Policies for Continuous Control
  5. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
  6. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
  7. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
  8. 2023Score Regularized Policy Optimization through Diffusion Behavior