# Block-Recurrent Dynamics in Vision Transformers

arXiv: https://arxiv.org/abs/2512.19941 · HTML: https://mlsanity.com/paper/2512.19941/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 99 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 30 of 72)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)

## Built on by (most cited first, 20 of 20)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Parcae: Scaling Laws For Stable Looped Language Models](https://mlsanity.com/paper/2604.12946/) (2026)
- [Tiny Autoregressive Recursive Models](https://mlsanity.com/paper/2603.08082/) (2026)
- [D'ej`a View: Looping Transformers for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2605.30215/) (2026)
- [Step-resolved data attribution for looped transformers](https://mlsanity.com/paper/2602.10097/) (2026)
- [Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology](https://mlsanity.com/paper/2605.14258/) (2026)
- [Pretraining Recurrent Networks without Recurrence](https://mlsanity.com/paper/2606.06479/) (2026)
- [Investigating Single-Block Recurrence in Vision Transformers for Image Recognition](https://mlsanity.com/paper/2605.10661/) (2026)
- [Human-level 3D shape perception emerges from multi-view learning](https://mlsanity.com/paper/2602.17650/) (2026)
- [Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision](https://mlsanity.com/paper/2608.04879/) (2026)
- [Programs-of-Layers in LLMs through the Lens of Cortical Areas](https://mlsanity.com/paper/2609.31360/) (2026)
- [Dynamic Mode Decomposition along Depth in Vision Transformers](https://mlsanity.com/paper/2605.07556/) (2026)
- [A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions](https://mlsanity.com/paper/2606.08105/) (2026)
- [Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers](https://mlsanity.com/paper/2608.00264/) (2026)
- [ProgResViT: Progressive Resolution and Width for Adaptive Vision Transformers](https://mlsanity.com/paper/2609.03216/) (2026)
- [LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking](https://mlsanity.com/paper/2609.33306/) (2026)
- [Procedural Core: A Compact Recurrent Initialization for Vision Transformers](https://mlsanity.com/paper/2609.37631/) (2026)
- [An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers](https://mlsanity.com/paper/2610.05558/) (2026)
- [Tabular Foundation Models Are Effectively Shallow](https://mlsanity.com/paper/2610.13253/) (2026)
- [Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers](https://mlsanity.com/paper/2610.16817/) (2026)

## Abstract (arXiv)

As Vision Transformers (ViTs) become standard vision backbones, a mechanistic account of their computational phenomenology is essential. Despite architectural cues that hint at dynamical structure, there is no settled framework that interprets Transformer depth as a well-characterized flow. In this work, we introduce the Block-Recurrent Hypothesis (BRH), arguing that trained ViTs admit a block-recurrent depth structure such that the computation of the original L blocks can be accurately rewritten using only k ll L distinct blocks applied recurrently. Across diverse ViTs, between-layer representational similarity matrices suggest few contiguous phases. To determine whether these phases reflect genuinely reusable computation, we train block-recurrent surrogates of pretrained ViTs: Recurrent Approximations to Phase-structured TransfORmers (Raptor). In small-scale, we demonstrate that stochastic depth and training promote recurrent structure and subsequently correlate with our ability to accurately fit Raptor. We then provide an empirical existence proof for BRH by training a Raptor model to recover 96% of DINOv2 ImageNet-1k linear probe accuracy in only 2 blocks at equivalent runtime. Finally, we leverage our hypothesis to develop a program of Dynamical Interpretability. We find i) directional convergence into class-dependent angular basins with self-correcting trajectories under small perturbations, ii) token-specific dynamics, where cls executes sharp late reorientations while patch tokens exhibit strong late-stage coherence toward their mean direction, and iii) a collapse to low rank updates in late depth, consistent with convergence to low-dimensional attractors. Altogether, we find a compact recurrent program emerges along ViT depth, pointing to a low-complexity normative solution that enables these models to be studied through principled dynamical systems analysis.