# Do large language vision models understand 3D shapes?

arXiv: https://arxiv.org/abs/2412.10908 · HTML: https://mlsanity.com/paper/2412.10908/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions](https://mlsanity.com/paper/2404.07214/) (2024)
- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information](https://mlsanity.com/paper/2412.00947/) (2024)
- [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023)
- [A General Protocol to Probe Large Vision Models for 3D Physical Understanding](https://mlsanity.com/paper/2310.06836/) (2023)
- [Language-Image Models with 3D Understanding](https://mlsanity.com/paper/2405.03685/) (2024)
- [Can We Talk Models Into Seeing the World Differently?](https://mlsanity.com/paper/2403.09193/) (2024)
- [Can 3D Vision-Language Models Truly Understand Natural Language?](https://mlsanity.com/paper/2403.14760/) (2024)
- [Approaching human 3D shape perception with neurally mappable models](https://mlsanity.com/paper/2308.11300/) (2023)
- [Vastextures: Vast repository of textures and PBR materials extracted from real-world images using unsupervised methods](https://mlsanity.com/paper/2406.17146/) (2024)

## Built on by (most cited first, 2 of 2)

- [Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation](https://mlsanity.com/paper/2506.21876/) (2025)
- [Shape and Texture Recognition in Large Vision-Language Models](https://mlsanity.com/paper/2503.23062/) (2025)

## Abstract (arXiv)

Large vision language models (LVLM) are the leading A.I approach for achieving a general visual understanding of the world. Models such as GPT, Claude, Gemini, and LLama can use images to understand and analyze complex visual scenes. 3D objects and shapes are the basic building blocks of the world, recognizing them is a fundamental part of human perception. The goal of this work is to test whether LVLMs truly understand 3D shapes by testing the models ability to identify and match objects of the exact same 3D shapes but with different orientations and materials/textures. A large number of test images were created using CGI with a huge number of highly diverse objects, materials, and scenes. The results of this test show that the ability of such models to match 3D shapes is significantly below humans but much higher than random guesses. Suggesting that the models have gained some abstract understanding of 3D shapes but still trail far beyond humans in this task. Mainly it seems that the models can easily identify the same object with a different orientation as well as matching identical 3D shapes of the same orientation but with different materials and textures. However, when both the object material and orientation are changed, all models perform poorly relative to humans. Code and benchmark are available.