# Geometric Parrots: Rule Absence vs. Rule Fragility in Spatial and Physical Reasoning for Vision-Language Models

arXiv: https://arxiv.org/abs/2610.16170 · HTML: https://mlsanity.com/paper/2610.16170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [Learning Physical Intuition of Block Towers by Example](https://mlsanity.com/paper/1603.01312/) (2016)
- [PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models](https://mlsanity.com/paper/2504.16074/) (2025)
- [Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive science](https://mlsanity.com/paper/2603.15381/) (2026)
- [VLDBench Evaluating Multimodal Disinformation with Regulatory Alignment](https://mlsanity.com/paper/2502.11361/) (2025)
- [Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs](https://mlsanity.com/paper/2607.03647/) (2026)
- [The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning](https://mlsanity.com/paper/2603.13372/) (2026)
- [Can Vision Language Models Learn Intuitive Physics from Interaction?](https://mlsanity.com/paper/2602.06033/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) can answer many geometric and physical reasoning questions correctly, yet still behave inconsistently in interaction, for example, abandoning a correct answer when confronted with a confident but incorrect user assertion. Standard single-turn accuracy cannot distinguish such capitulations from beneficial corrections because it scores each response in isolation. We introduce GRIP, a controlled benchmark with 34 procedurally generated reasoning domains, 100,000 images, and 600,000 questions spanning geometric, spatial, visual, and physical reasoning. Each scene is generated by deterministic code, and every answer is derived directly from the underlying scene state, providing exact ground truth without human annotation. On GRIP, we propose Dual-Loop Evaluation. A closed loop probes each image with a five-step compositional question ladder to localize where reasoning fails. An open loop then re-elicits the model's answer under controlled challenges, including peer-model responses and unsupported user assertions. Because ground truth is known exactly, answer changes can be signed as corrections (wrong-to-right) or capitulations (right-to-wrong), separating two failure modes: rule absence and rule fragility. The dataset is available at https://huggingface.co/datasets/bilaljawaid980/GRIP-Benchmark, with further details at https://bilaljawaid980.github.io/grip-benchmark/.