mlsanityData to 7 Oct 2026

Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

arXiv 2609.28838 ↗First posted 23 Sep 2026cs.RO cs.LGcited by 0

On the map: Vision-language-action models → Reinforcement learning and post-training

also in 1 more
Abstract (arXiv)

Online reinforcement learning fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates often destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO with stochastic (SDE) sampling. Three exploration-noise policies differ in one live variable: a fixed noise scale, a ReinFlow-style learned noise network, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement shows the controller's action expert keeps changing, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result, not offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release tools measuring per-task collapse under four definitions, rescoring noise and instrument tares.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 48 of its 51 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

48 of 51 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models29 Jun
  2. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning12 Mar
  3. 2025Reinforcing Action Policies by Prophesying25 Nov
  4. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models3 Nov
  5. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models8 Oct
  6. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning11 Sep
  7. 2025What Can RL Bring to VLA Generalization? An Empirical Study26 May
  8. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning24 May
  9. 2025Interactive Post-Training for Vision-Language-Action Models22 May
  10. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy8 Feb
  11. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning28 Jan

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  9. 2017Deep Reinforcement Learning that Matterscited by 727
  10. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  11. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
25 more
  1. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  2. 2024Diffusion Policy Policy Optimizationcited by 255
  3. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  4. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  5. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  6. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  7. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  8. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  9. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  10. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 51
  11. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 45
  12. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 36
  13. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 31
  14. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  15. 2025Verifier-free Test-Time Sampling for Vision-Language-Action Modelscited by 21
  16. 2020Revisiting BFloat16 Trainingcited by 15
  17. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  18. 2026Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RLcited by 5
  19. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 5
  20. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  21. 2026Tri-Info: Generalizable, Interpretable Failure Prediction for VLA Models via Information Theorycited by 3
  22. 2026VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Modelscited by 1
  23. 2026DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuningcited by 1
  24. 2026Dual-Flow Reinforcement Learning with State-Aware Explorationcited by 1
  25. 2026Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policiescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
  2. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
  3. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation
  4. 2026GroundControl: Anticipating Navigation Failures in Vision-Language Agents via Trajectory-Consistent Uncertainty Estimates
  5. 2025Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach
  6. 2026T2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning
  7. 2026When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering
  8. 2026SmoothVLA: Aligning Vision-Language-Action Models with Physical Constraints via Intrinsic Smoothness Optimization