mlsanityData to 7 Oct 2026

Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policies

arXiv 2606.08015 ↗First posted 6 Jun 2026cs.ROcited by 2

On the map: Vision-language-action models → Efficient VLA inference and adaptation

also in 2 more
Abstract (arXiv)

We propose Q-Guided Value-Gradient Matching (Q-VGM), an offline-to-online RL method for fine-tuning flow-matching vision-language-action policies with a learned critic. Directly applying critic gradients to flow policies requires backpropagation through the multi-step denoising process (BPTT), which is costly and unstable at VLA scale. Motivated by an optimal-control view of denoising, we derive a local approximation connecting clean-action value gradients to local velocity corrections at late denoising steps. Q-VGM uses this connection to construct critic-guided targets for fine-tuning the pretrained action expert through velocity matching. Gradients pass only through local velocity predictions, avoiding backpropagation through the denoising trajectory. An action-sensitive chunk critic is trained with IQL offline and TD learning online. On LIBERO, Q-VGM improves the few-shot-SFT four-suite average from 84.6% to 90.7% offline using 150 policy rollouts per suite, and reaches 98.4% after online training. Compared with on-policy RL, Q-VGM achieves 5.3x higher sample efficiency on average across four LIBERO suites, measured by the episode budget for 95% rollout success rate. On three real-world bimanual tasks, offline Q-VGM improves average success from 66.7% to 98.3%.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Efficient VLA inference and adaptation, one of 13 directions in Vision-language-action models. Builds on 44 of its 49 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

44 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models11 Oct
  2. 2025Verifier-free Test-Time Sampling for Vision-Language-Action Models7 Oct
  3. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models21 Jun
  4. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Models16 Jan

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
28 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  8. 2024Diffusion Policy Policy Optimizationcited by 255
  9. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  10. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  11. 2025Flow Q-Learningcited by 162
  12. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  13. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  14. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  15. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  16. 2025Reinforcement Learning with Action Chunkingcited by 97
  17. 2025Flow Matching Policy Gradientscited by 86
  18. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  19. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  20. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  21. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  22. 2025Horizon Reduction Makes RL Scalablecited by 55
  23. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  24. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  25. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
  26. 2026Q-learning with Adjoint Matchingcited by 33
  27. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  28. 2026Trust Region Q Adjoint Matchingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 3
  2. 2026Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interfacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
  2. 2026Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT
  3. 2026SnapFlow: One-Step Action Generation for Flow-Matching VLAs via Progressive Self-Distillation
  4. 2026Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement
  5. 2026Guided Action Flow: Q-Guided Inference for Flow-Matching Vision-Language-Action Policies
  6. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models
  7. 2026FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization
  8. 2026Fewer Steps, Better Actions: Rethinking Flow-Matching Inference for VLA Policies