# Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models

arXiv: https://arxiv.org/abs/2601.12626 · HTML: https://mlsanity.com/paper/2601.12626/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 18 Jan 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens](https://mlsanity.com/paper/2411.16724/) (2024)
- [GRIT: Teaching MLLMs to Think with Images](https://mlsanity.com/paper/2505.15879/) (2025)
- [Towards Interpreting Visual Information Processing in Vision-Language Models](https://mlsanity.com/paper/2410.07149/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas](https://mlsanity.com/paper/2503.01773/) (2025)
- [Monitoring Latent World States in Language Models with Propositional Probes](https://mlsanity.com/paper/2406.19501/) (2024)
- [From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks](https://mlsanity.com/paper/2406.06579/) (2024)
- [Temporal Reasoning Transfer from Text to Video](https://mlsanity.com/paper/2410.06166/) (2024)
- [Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2503.19707/) (2025)
- [Is CLIP ideal? No. Can we fix it? Yes!](https://mlsanity.com/paper/2503.08723/) (2025)
- [Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering](https://mlsanity.com/paper/2411.10950/) (2024)

## Built on by (most cited first, 14 of 14)

- [Pathways of Visual Information Flow in Vision-Language Models](https://mlsanity.com/paper/2607.03358/) (2026)
- [The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models](https://mlsanity.com/paper/2603.22278/) (2026)
- [Why Far Looks Up: Probing Spatial Representation in Vision-Language Models](https://mlsanity.com/paper/2605.30161/) (2026)
- [3D Primitives are a Spatial Language for VLMs](https://mlsanity.com/paper/2605.12586/) (2026)
- [Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning](https://mlsanity.com/paper/2606.31257/) (2026)
- [Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models](https://mlsanity.com/paper/2605.07148/) (2026)
- [Interaction Locality in Hierarchical Recursive Reasoning](https://mlsanity.com/paper/2605.20784/) (2026)
- [Knowing Isn't Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?](https://mlsanity.com/paper/2608.22916/) (2026)
- [TrianguLang: Geometry-Aware Semantic Consensus for Pose-Free 3D Localization](https://mlsanity.com/paper/2603.08096/) (2026)
- [On Temporal Binding in Large Audio Language Models](https://mlsanity.com/paper/2609.34806/) (2026)
- [Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning](https://mlsanity.com/paper/2609.35486/) (2026)
- [Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs](https://mlsanity.com/paper/2610.05715/) (2026)
- [S-Space: Exploring Spatial Workspace in Multimodal Language Models](https://mlsanity.com/paper/2610.13527/) (2026)
- [Seeing Depth but Choosing Cues in Vision-Language Models](https://mlsanity.com/paper/2610.17976/) (2026)

## Abstract (arXiv)

Spatio-temporal reasoning is a remarkable capability of Vision Language Models (VLMs), but the underlying mechanisms of such abilities remain largely opaque. We postulate that visual/geometrical and textual representations of spatial structure must be combined at some point in VLM computations. We search for such confluence, and ask whether the identified representation can causally explain aspects of input-output model behavior through a linear model. We show empirically that VLMs encode object locations by linearly binding textitspatial IDs to textual activations, then perform reasoning via language tokens. Through rigorous causal interventions we demonstrate that these IDs, which are ubiquitous across the model, can systematically mediate model beliefs at intermediate VLM layers. Additionally, we find that spatial IDs serve as a diagnostic tool for identifying limitations in existing VLMs, and as a valuable learning signal. We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism. By characterizing our proposed spatiotemporal ID mechanism, we elucidate a previously underexplored internal reasoning process in VLMs, toward improved interpretability and the principled design of more aligned and capable models. We release our code for reproducibility: https://github.com/Raphoo/linear-mech-vlms.