# Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules

arXiv: https://arxiv.org/abs/2512.10300 · HTML: https://mlsanity.com/paper/2512.10300/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning](https://mlsanity.com/paper/2208.05358/) (2022)
- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning](https://mlsanity.com/paper/2212.00259/) (2022)
- [Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2005.07310/) (2020)
- [Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding](https://mlsanity.com/paper/2503.06287/) (2025)
- [On the Role of Attention Heads in Large Language Model Safety](https://mlsanity.com/paper/2410.13708/) (2024)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition](https://mlsanity.com/paper/1406.3284/) (2014)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [Core Knowledge Deficits in Multi-Modal Language Models](https://mlsanity.com/paper/2410.10855/) (2024)
- [Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach](https://mlsanity.com/paper/2412.18108/) (2024)
- [Cognitive Mirrors: Exploring the Diverse Functional Roles of Attention Heads in LLM Reasoning](https://mlsanity.com/paper/2512.10978/) (2025)

## Built on by (most cited first, 2 of 2)

- [MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues](https://mlsanity.com/paper/2605.21954/) (2026)
- [When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities](https://mlsanity.com/paper/2607.08605/) (2026)

## Abstract (arXiv)

Despite excelling on multimodal benchmarks, vision-language models (VLMs) largely remain a black box. In this paper, we propose a novel interpretability framework to systematically analyze the internal mechanisms of VLMs, focusing on the functional roles of attention heads in multimodal reasoning. To this end, we introduce CogVision, a dataset that decomposes complex multimodal questions into step-by-step subquestions designed to simulate human reasoning through a chain-of-thought paradigm, with each subquestion associated with specific receptive or cognitive functions such as high-level visual reception and inference. Using a probing-based methodology, we identify attention heads that specialize in these functions and characterize them as functional heads. Our analysis across diverse VLM families reveals that these functional heads are universally sparse, vary in number and distribution across functions, and mediate interactions and hierarchical organization. Furthermore, intervention experiments demonstrate their critical role in multimodal reasoning: removing functional heads leads to performance degradation, while emphasizing them enhances accuracy. These findings provide new insights into the cognitive organization of VLMs and suggest promising directions for designing models with more human-aligned perceptual and reasoning abilities.