mlsanityData to 7 Oct 2026

From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment

arXiv 2502.01828 ↗First posted 3 Feb 2025cs.RO cs.LGcited by 51

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

While generative robot policies have demonstrated significant potential in learning complex, multimodal behaviors from demonstrations, they still exhibit diverse failures at deployment-time. Policy steering offers an elegant solution to reducing the chance of failure by using an external verifier to select from low-level actions proposed by an imperfect generative policy. Here, one might hope to use a Vision Language Model (VLM) as a verifier, leveraging its open-world reasoning capabilities. However, off-the-shelf VLMs struggle to understand the consequences of low-level robot actions as they are represented fundamentally differently than the text and images the VLM was trained on. In response, we propose FOREWARN, a novel framework to unlock the potential of VLMs as open-vocabulary verifiers for runtime policy steering. Our key idea is to decouple the VLM's burden of predicting action outcomes (foresight) from evaluation (forethought). For foresight, we leverage a latent world model to imagine future latent states given diverse low-level action plans. For forethought, we align the VLM with these predicted latent states to reason about the consequences of actions in its native representation--natural language--and effectively filter proposed plans. We validate our framework across diverse robotic manipulation tasks, demonstrating its ability to bridge representational gaps and provide robust, generalizable policy steering. Videos can be found on the project website: https://yilin-wu98.github.io/forewarn/.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 51 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

37 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation1 Oct

Other references · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2023Mastering Diverse Domains through World Modelscited by 1,193
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  11. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  12. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
24 more
  1. 2017Inverse Reward Designcited by 246
  2. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  3. 2024Behavior Generation with Latent Actionscited by 159
  4. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  5. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  6. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  7. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  8. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  9. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  10. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  11. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  12. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  13. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  14. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  15. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  16. 2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
  17. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  18. 2024Hybrid Inverse Reinforcement Learningcited by 31
  19. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  20. 2024How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluationcited by 14
  21. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  22. 2024I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiencescited by 11
  23. 2016On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Trainingcited by 9
  24. 2024Future Success Prediction in Open-Vocabulary Object Manipulation Tasks Based on End-Effector Trajectoriescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 28.

51 papers.

Sort:

By area:

  1. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  7. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 27
  8. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  9. 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verificationcited by 25
  10. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoningcited by 21
  11. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  12. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
38 more
  1. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  2. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  3. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibilitycited by 16
  4. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  5. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Modelscited by 14
  6. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  7. 2026Foundational World Models Accurately Detect Bimanual Manipulator Failurescited by 11
  8. 2025Latent Policy Steering with Embodiment-Agnostic Pretrained World Modelscited by 8
  9. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusioncited by 8
  10. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  11. 2026RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulationcited by 7
  12. 2026Rewind-IL: Online Failure Detection and State Respawning for Imitation Learningcited by 7
  13. 2025Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Controlcited by 6
  14. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
  15. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policiescited by 5
  16. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  17. 2026Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimizationcited by 4
  18. 2026When to Act, Ask, or Learn: Uncertainty-Aware Policy Steeringcited by 4
  19. 2025Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondencecited by 3
  20. 2026TEXEDO : Test Time Scaling for Controller-aware Language-conditioned Humanoid Motion Generationcited by 3
  21. 2025EVE: A Generator-Verifier System for Generative Policiescited by 2
  22. 2026PPGuide: Steering Diffusion Policies with Performance Predictive Guidancecited by 2
  23. 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrailscited by 1
  24. 2026CulinaryCut-VLAP: A Vision-Language-Action-Physics Framework for Food Cutting via a Force-Aware Material Point Methodcited by 1
  25. 2026Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Modelscited by 1
  26. 2026Grounded World Model for Semantically Generalizable Planningcited by 1
  27. 2026SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLAcited by 1
  28. 2026Visual Verification Enables Inference-time Steering and Autonomous Policy Improvementcited by 1
  29. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  30. 2026StressDream: Steering Video World Models for Robust Policy Evaluation and Improvementcited by 0
  31. 2026VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoringcited by 0
  32. 2026DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuningcited by 0
  33. 2026DreamLedger: Where to Refuse World-Model Imagination Using Execution-Settled Creditcited by 0
  34. 2026Training-Free Action Correction for VLA Model Failures via Language Feedbackcited by 0
  35. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  36. 2026Beyond Visual Quality: A Study of Test-Time Planning with World Action Modelscited by 0
  37. 2026GT-VLA: Target-Conditioned Trace Guidance for Generalizable Robotic Manipulationcited by 0
  38. 2026Cue the Flow: Steering Flow-Matching Policies for Open-World Delivery Manipulationcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization
  2. 2026SLIP-VLA: Single-Step Latent Imagination for Policy Learning in Vision-Language-Action Models
  3. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
  4. 2026AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies
  5. 2026ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
  6. 2026Reasoning Without Inference Cost: Latent Semantic Scaffolding for Robot VLA Policies
  7. 2026AcrossWAM1.0:A Modular Latent World-Action Stack for Compact Robot Policies
  8. 2026IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models