mlsanityData to 7 Oct 2026

Do large language vision models understand 3D shapes?

arXiv 2412.10908 ↗First posted 14 Dec 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision language models (LVLM) are the leading A.I approach for achieving a general visual understanding of the world. Models such as GPT, Claude, Gemini, and LLama can use images to understand and analyze complex visual scenes. 3D objects and shapes are the basic building blocks of the world, recognizing them is a fundamental part of human perception. The goal of this work is to test whether LVLMs truly understand 3D shapes by testing the models ability to identify and match objects of the exact same 3D shapes but with different orientations and materials/textures. A large number of test images were created using CGI with a huge number of highly diverse objects, materials, and scenes. The results of this test show that the ability of such models to match 3D shapes is significantly below humans but much higher than random guesses. Suggesting that the models have gained some abstract understanding of 3D shapes but still trail far beyond humans in this task. Mainly it seems that the models can easily identify the same object with a different orientation as well as matching identical 3D shapes of the same orientation but with different materials and textures. However, when both the object material and orientation are changed, all models perform poorly relative to humans. Code and benchmark are available.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 24 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

20 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Can We Talk Models Into Seeing the World Differently?14 Mar
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions20 Feb
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  6. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  8. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  9. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  10. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  11. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  12. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 33
5 more
  1. 2023A General Protocol to Probe Large Vision Models for 3D Physical Understandingcited by 32
  2. 2024Language-Image Models with 3D Understandingcited by 31
  3. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  4. 2023Approaching human 3D shape perception with neurally mappable modelscited by 6
  5. 2024Vastextures: Vast repository of textures and PBR materials extracted from real-world images using unsupervised methodscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  2. 2025Shape and Texture Recognition in Large Vision-Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
  2. 2024Do Vision-Language Models Really Understand Visual Language?
  3. 2026VLM3: Vision Language Models Are Native 3D Learners
  4. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models
  5. 2024Vision language models are blind: Failing to translate detailed visual features into words
  6. 2024Understanding Depth and Height Perception in Large Visual-Language Models
  7. 2024Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities
  8. 2024Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models