mlsanityData to 7 Oct 2026

Physically Grounded Vision-Language Models for Robotic Manipulation

arXiv 2309.02561 ↗First posted 5 Sep 2023cs.RO cs.AI cs.CVcited by 163

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically grounded VLMs. We additionally illustrate the benefits of our physically grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 87 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

37 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
24 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  6. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  7. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  8. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  9. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  10. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  11. 2023PACO: Parts and Attributes of Common Objectscited by 146
  12. 2023Vision-Language Models as Success Detectorscited by 138
  13. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  14. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  15. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  16. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  17. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  18. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  19. 2020Visual Grounding of Learned Physical Modelscited by 54
  20. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  21. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 42
  22. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  23. 2023Can Language Models Understand Physical Concepts?cited by 19
  24. 2023Semantic Mechanical Search with Large Vision and Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 63; 2025: 63; 2026: 28.

163 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  3. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  4. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  6. 2023Large Language Models for Robotics: A Surveycited by 132
  7. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  8. 2024Vision Language Models are In-Context Value Learnerscited by 84
  9. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  10. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  11. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  12. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
38 more
  1. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  2. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  3. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
  4. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  5. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  6. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  7. 2024AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulationcited by 52
  8. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  9. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  10. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  11. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulationcited by 40
  12. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  13. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  14. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  15. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  16. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28
  17. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  18. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Modelscited by 25
  19. 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verificationcited by 25
  20. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
  21. 2025Generative Physical AI in Vision: A Surveycited by 24
  22. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  23. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24
  24. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  25. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  26. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Frameworkcited by 20
  27. 2025A Survey on Efficient Vision-Language Modelscited by 19
  28. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
  29. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  30. 2024Large Action Models: From Inception to Implementationcited by 18
  31. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answeringcited by 15
  32. 20243D Part Segmentation via Geometric Aggregation of 2D Visual Featurescited by 15
  33. 2025RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulationcited by 15
  34. 2024PIN: Positional Insert Unlocks Object Localisation Abilities in VLMscited by 14
  35. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  36. 2024Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulationcited by 13
  37. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agentscited by 13
  38. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12

50 most cited of 163 All 163 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  2. 2025Physically Ground Commonsense Knowledge for Articulated Object Manipulation with Analytic Concepts
  3. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  4. 2025PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?
  5. 2026RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
  6. 2026PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models
  7. 2023Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning
  8. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting