Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Despite the importance of shape perception in human vision, early neural image classifiers relied less on shape information for object recognition than other (often spurious) features. While recent research suggests that current large Vision-Language Models (VLMs) exhibit more reliance on shape, we find them to still be seriously limited in this regard. To quantify such limitations, we introduce IllusionBench, a dataset that challenges current cutting-edge VLMs to decipher shape information when the shape is represented by an arrangement of visual elements in a scene. Our extensive evaluations reveal that, while these shapes are easily detectable by human annotators, current VLMs struggle to recognize them, indicating important avenues for future work in developing more robust visual perception systems. The full dataset and codebase are available at: urlhttps://arshiahemmat.github.io/illusionbench/
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 76 references that are in the map. First posted on arXiv 9 Nov 2024.
Builds on
19 of 76 references.
Closest ancestors · 1
Same direction, newest first
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
Other references · 18
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2023GPT-4 Technical Reportcited by 18,132
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Qwen Technical Reportcited by 3,273
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2020In Search of Lost Domain Generalizationcited by 1,042
- 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
- 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
6 more
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
- 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
- 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
- 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
- 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
- 2023Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalizationcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models
- 2024GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models
- 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
- 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
- 2025Shape and Texture Recognition in Large Vision-Language Models
- 2026The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
- 2026ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
- 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models