mlsanityData to 14 Oct 2026

Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models

arXiv 2607.16506 ↗First posted 17 Jul 2026cs.RO cs.LGcited by 2

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills. We show this failure mode in residual reinforcement learning (RL) over a frozen VLA base policy: constant sparse success rewards improve each subtask in isolation yet yield little or no gain when skills are chained, because terminal state quality is uncontrolled. We propose Foresight Residual RL, which optimizes handoff quality by augmenting each subtask's sparse success reward with an offline-estimated foresight value -- the probability of future subtask success conditioned on the terminal state of the current subtask. Concretely, we (i) train a visual foresight predictor from images of terminal states of the base policy, labeled using downstream rollout statistics, and (ii) train residual policies via backward foresight induction, using the predictor output as a reward multiplier. On a three-phase wrench-based nut-tightening assembly task in Isaac Gym (grasp, move-insert, rotate), our method achieves 85.6% full-task success, outperforming standard subtask residual RL (54.5%) and VLA baselines, while leaving per-subtask success unchanged. These results highlight that improving long-horizon performance requires shaping which successful states are produced at each sub-task, not only whether success occurs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 23 of its 31 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

23 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RL30 Oct
  2. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  5. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  6. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  7. 2016The Option-Critic Architecturecited by 448
  8. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  9. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  10. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  11. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  12. 2018Residual Policy Learningcited by 164
9 more
  1. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  2. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  3. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
  4. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  5. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  6. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 46
  7. 2022Reward-Respecting Subtasks for Model-Based Reinforcement Learningcited by 16
  8. 2025Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policiescited by 2
  9. 2024Composing Option Sequences by Adaptation: Initial Resultscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memorycited by 5
  2. 2026RouteRLT: Learning When and Which RL Specialist Should Control a Vision-Language-Action Policycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RL
  2. 2026Learning More from Less: Reinforcement Learning from Hindsight
  3. 2026Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning
  4. 2026From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention
  5. 2025Training-free Generation of Temporally Consistent Rewards from VLMs
  6. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
  7. 2021Residual Reinforcement Learning from Demonstrations
  8. 2026StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks