mlsanityData to 7 Oct 2026

VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation

arXiv 2407.09829 ↗First posted 13 Jul 2024cs.ROcited by 32

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Although Model Predictive Control (MPC) can effectively predict the future states of a system and thus is widely used in robotic manipulation tasks, it does not have the capability of environmental perception, leading to the failure in some complex scenarios. To address this issue, we introduce Vision-Language Model Predictive Control (VLMPC), a robotic manipulation framework which takes advantage of the powerful perception capability of vision language model (VLM) and integrates it with MPC. Specifically, we propose a conditional action sampling module which takes as input a goal image or a language instruction and leverages VLM to sample a set of candidate action sequences. Then, a lightweight action-conditioned video prediction model is designed to generate a set of future frames conditioned on the candidate action sequences. VLMPC produces the optimal action sequence with the assistance of VLM through a hierarchical cost function that formulates both pixel-level and knowledge-level consistence between the current observation and the goal image. We demonstrate that VLMPC outperforms the state-of-the-art methods on public benchmarks. More importantly, our method showcases excellent performance in various real-world tasks of robotic manipulation. Code is available at~urlhttps://github.com/PPjmchen/VLMPC.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 77 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

60 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration20 Nov
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  7. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  11. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  12. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  13. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  14. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  15. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  16. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  17. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  18. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  19. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  20. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  21. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  22. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  23. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  24. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  25. 2023Video Language Planningcited by 160
  26. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  27. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  28. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  29. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  30. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  31. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  32. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  33. 2019Deep Visual MPC-Policy Learning for Navigationcited by 54
  34. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  35. 2021STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulationcited by 46
  36. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  37. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  38. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  2. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
  3. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  4. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  5. 2024Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulationcited by 13
  6. 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 10
  7. 2026Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policiescited by 8
  8. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  9. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  10. 2026PRISM: PRior-guided Imagination Sampling in world Modelscited by 7
  11. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  12. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 4
20 more
  1. 2025Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generationcited by 4
  2. 2025Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interfacecited by 4
  3. 2026Medical world models: representing medical states, modelling clinical dynamics and guiding intervention policiescited by 3
  4. 2026AdaReP:Adaptive Re-Planning under Model Mismatch for Neural World-Model Predictive Controlcited by 3
  5. 2025TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Modelscited by 2
  6. 2025VERM: Leveraging Foundation Models to Create a Virtual Eye for Efficient 3D Robotic Manipulationcited by 2
  7. 2026AnoleVLA: Lightweight Vision-Language-Action Model with Deep State Space Models for Mobile Manipulationcited by 2
  8. 2026BadWAM: When World-Action Models Dream Right but Act Wrongcited by 2
  9. 2026TrAct: Bridging Robot Control and Visual Prediction with Visual Trackscited by 2
  10. 2025T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Modelscited by 1
  11. 2025Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Modelcited by 1
  12. 2026HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wildcited by 1
  13. 2026CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulationcited by 1
  14. 2026Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAscited by 1
  15. 2025AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulationcited by 0
  16. 2025Fake or Real, Can Robots Tell? Evaluating VLM Robustness to Domain Shift in Single-View Robotic Scene Understandingcited by 0
  17. 2025FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Modelscited by 0
  18. 2026Event-Adaptive Motion Planning with Distilled Vision-Language Model in Safety-Critical Situationscited by 0
  19. 2026From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigationcited by 0
  20. 2026Language-Guided Terrain-Adaptive Neural MPC for Autonomous Traversal of Articulated Tracked Robotscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  2. 2025Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
  3. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  4. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
  5. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
  6. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
  7. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation
  8. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review