mlsanityData to 14 Oct 2026

Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning

arXiv 2311.17842 ↗First posted 29 Nov 2023cs.RO cs.AI cs.CLcited by 161

Not on the map. Why?

Cited by 50 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this study, we are interested in imbuing robots with the capability of physically-grounded task planning. Recent advancements have shown that large language models (LLMs) possess extensive knowledge useful in robotic tasks, especially in reasoning and planning. However, LLMs are constrained by their lack of world grounding and dependence on external affordance models to perceive environmental information, which cannot jointly reason with LLMs. We argue that a task planner should be an inherently grounded, unified multimodal system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a novel approach for long-horizon robotic planning that leverages vision-language models (VLMs) to generate a sequence of actionable steps. ViLa directly integrates perceptual data into its reasoning and planning process, enabling a profound understanding of commonsense knowledge in the visual world, including spatial layouts and object attributes. It also supports flexible multimodal goal specification and naturally incorporates visual feedback. Our extensive evaluation, conducted in both real-robot and simulated environments, demonstrates ViLa's superiority over existing LLM-based planners, highlighting its effectiveness in a wide array of open-world manipulation tasks.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Builds on 80 of its 97 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

80 of 97 references.

References · 80

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  13. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  15. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  16. 2019How Can We Know What Language Models Know?cited by 959
  17. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  18. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  19. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  20. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  21. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  22. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  23. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  24. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  25. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  26. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  27. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  28. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  31. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  32. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  33. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  34. 2023Language Models Represent Space and Timecited by 316
  35. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  36. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  37. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  38. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 61; 2025: 71; 2026: 29.

161 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  2. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  3. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  4. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  5. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 147
  6. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
  7. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  8. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 121
  9. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  10. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 107
  11. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 99
  12. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 86
38 more
  1. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  2. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  3. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  4. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 69
  5. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  6. 2024Human-Object Interaction from Human-Level Instructionscited by 64
  7. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 58
  8. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
  9. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  10. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  11. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  12. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  13. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  14. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planningcited by 39
  15. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 38
  16. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  17. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  18. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 33
  19. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 33
  20. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 32
  21. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 32
  22. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  23. 2025dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thoughtcited by 30
  24. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28
  25. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 27
  26. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 26
  27. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 25
  28. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  29. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  30. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  31. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  32. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21
  33. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  34. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 20
  35. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 20
  36. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 20
  37. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 19
  38. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
  2. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  3. 2024RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
  4. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning
  5. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  6. 2026APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts
  7. 20253D-Grounded Vision-Language Framework for Robotic Task Planning: Automated Prompt Synthesis and Supervised Reasoning
  8. 2025Vision-Language-Policy Model for Dynamic Robot Task Planning