mlsanityData to 14 Oct 2026

Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?

arXiv 2510.16924 ↗First posted 19 Oct 2025cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Despite significant progress in multimodal language models (LMs), it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. To address this question, we propose a novel embodied knowledge understanding benchmark based on the perceptual theory from psychology, encompassing visual, auditory, tactile, gustatory, olfactory external senses, and interoception. The benchmark assesses the models' perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. By comparing 30 state-of-the-art LMs, we surprisingly find that vision-language models (VLMs) do not outperform text-only models in either task. Moreover, the models perform significantly worse in the visual dimension compared to other sensory dimensions. Further analysis reveals that the vector representations are easily influenced by word form and frequency, and the models struggle to answer questions involving spatial perception and reasoning. Our findings underscore the need for more effective integration of embodied knowledge in LMs to enhance their understanding of the physical world.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 64 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

26 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
11 more
  1. 2020Experience Grounds Languagecited by 242
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  3. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  4. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  5. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 69
  6. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  7. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  8. 2024Octopi: Object Property Reasoning with Large Tactile-Language Modelscited by 48
  9. 2021Does Vision-and-Language Pretraining Improve Lexical Grounding?cited by 25
  10. 2023Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representationscited by 4
  11. 2020Intuitive Contrasting Map for Antonym Embeddingscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026How does fine-tuning improve sensorimotor representations in large language models?cited by 0
  2. 2026Tanawpamana: Benchmarking Vision-Language Models on Philippine Cultural Heritage Sitescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
  2. 2023Can Language Models Understand Physical Concepts?
  3. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
  4. 2023GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models
  5. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models
  6. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  7. 2025Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings
  8. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding