# Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance

arXiv: https://arxiv.org/abs/2604.01848 · HTML: https://mlsanity.com/paper/2604.01848/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [Does Spatial Cognition Emerge in Frontier Models?](https://mlsanity.com/paper/2410.06468/) (2024)
- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [BabyVision: Visual Reasoning Beyond Language](https://mlsanity.com/paper/2601.06521/) (2026)
- [PACS: A Dataset for Physical Audiovisual CommonSense Reasoning](https://mlsanity.com/paper/2203.11130/) (2022)
- [Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?](https://mlsanity.com/paper/2410.19546/) (2024)
- [Reviving Shift Equivariance in Vision Transformers](https://mlsanity.com/paper/2306.07470/) (2023)
- [Position: Reasoning After Perception Means Reasoning Without Vision](https://mlsanity.com/paper/2507.16863/) (2025)
- [Visually Prompted Benchmarks Are Surprisingly Fragile](https://mlsanity.com/paper/2512.17875/) (2025)
- [RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation](https://mlsanity.com/paper/2508.13968/) (2025)
- [Large Vision Models Can Solve Mental Rotation Problems](https://mlsanity.com/paper/2509.15271/) (2025)

## Abstract (arXiv)

This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.