mlsanityData to 7 Oct 2026

Self-Improving Vision-Language-Action Models with Data Generation via Residual RL

arXiv 2511.00091 ↗First posted 30 Oct 2025cs.CV cs.ROcited by 72

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-language-action (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. We propose Probe, Learn, Distill (PLD), a three-stage plug-and-play framework that improves VLAs through residual reinforcement learning (RL) and distribution-aware data collection. In Stage 1, we train lightweight residual actors to probe failure regions of the VLA generalist. In Stage 2, we use a hybrid rollout scheme that aligns collected trajectories with the generalist's deployment distribution while capturing recovery behaviors. In Stage 3, we distill the curated trajectories back into the generalist with standard SFT. PLD achieves near-saturated 99% task success on LIBERO, over 50% gains in SimplerEnv, and 100% success on real-world Franka and YAM arm manipulation tasks. Ablations show that residual probing and distribution-aware replay are key to collecting deployment-aligned data that improves both seen and unseen tasks, offering a scalable path toward self-improving VLA models.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 53 of its 58 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

53 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning24 May
  2. 2025Interactive Post-Training for Vision-Language-Action Models22 May
  3. 2025Refined Policy Distillation: From VLA Generalists to RL Experts6 Mar
  4. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy8 Feb
  5. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2016Layer Normalizationcited by 5,579
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
36 more
  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  11. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  13. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  14. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  15. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  16. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  17. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  18. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  19. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  20. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  21. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  22. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  23. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  24. 2022Jump-Start Reinforcement Learningcited by 102
  25. 2025Reinforcement Learning with Action Chunkingcited by 97
  26. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  27. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  28. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  29. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  30. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  31. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  32. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  33. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28
  34. 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Owncited by 16
  35. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  36. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 68.

72 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  3. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  4. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  5. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  6. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 31
  7. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  8. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  9. 2025Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfercited by 13
  10. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
  11. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  12. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 9
38 more
  1. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  2. 2026RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Trainingcited by 7
  3. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  4. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  5. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  6. 2026IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulationcited by 5
  7. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptationcited by 5
  8. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  9. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  10. 2026When to Act, Ask, or Learn: Uncertainty-Aware Policy Steeringcited by 4
  11. 2026TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulationcited by 4
  12. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  13. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  14. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
  15. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 3
  16. 2026Trust Region Q Adjoint Matchingcited by 3
  17. 2026WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFTcited by 3
  18. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  19. 2026Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Modelscited by 3
  20. 2026DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimizationcited by 2
  21. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  22. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  23. 2026Robot Self-Improvement via Human-Video Dynamics Modelscited by 2
  24. 2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulationcited by 2
  25. 2026InSight: Self-Guided Skill Acquisition via Steerable VLAscited by 2
  26. 2026ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuningcited by 2
  27. 2026Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Modelscited by 2
  28. 2026How to Peel with a Knife: Aligning Fine-Grained Manipulation with Human Preferencecited by 1
  29. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  30. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 1
  31. 20263DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-trainingcited by 1
  32. 2026Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Modelscited by 1
  33. 2026Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancementcited by 1
  34. 2026Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steeringcited by 1
  35. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  36. 2026HALO-WA: Hybrid-Attention Latent-Guided Online Reinforcement Learning for World-Action Modelscited by 1
  37. 2026CLIFT: Turning Gemini Robotics On-Device into Humanoid Specialists via Non-Invasive Closed-Loop Iterative Fine-Tuningcited by 1
  38. 2026You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvementcited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models
  2. 2025Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach
  3. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
  4. 2026dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
  5. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
  6. 2026FiberTune: Preserving Action-Fiber Visual Residuals in Vision-Language-Action Fine-Tuning
  7. 2025What Can RL Bring to VLA Generalization? An Empirical Study
  8. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators