# Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex

arXiv: https://arxiv.org/abs/2609.36366 · HTML: https://mlsanity.com/paper/2609.36366/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 71 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [The neuroconnectionist research programme](https://mlsanity.com/paper/2209.03718/) (2022)
- [Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models](https://mlsanity.com/paper/2306.03089/) (2023)
- [Neural system identification for large populations separating "what" and "where"](https://mlsanity.com/paper/1711.02653/) (2017)
- [The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies](https://mlsanity.com/paper/2501.00504/) (2024)
- [Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity](https://mlsanity.com/paper/2405.03280/) (2024)
- [Transformer brain encoders explain human high-level visual responses](https://mlsanity.com/paper/2505.17329/) (2025)
- [Reanimating Images using Neural Representations of Dynamic Stimuli](https://mlsanity.com/paper/2406.02659/) (2024)
- [In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain](https://mlsanity.com/paper/2510.21142/) (2025)
- [Only Brains Align with Brains: Cross-Region Alignment Patterns Expose Limits of Normative Models](https://mlsanity.com/paper/2604.21780/) (2026)
- [NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity](https://mlsanity.com/paper/2607.02317/) (2026)

## Abstract (arXiv)

Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the majority of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding models address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fitting this model to fMRI responses to short video clips. We compare joint spatiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides interpretable, stimulus-specific attention maps that dynamically follow moving objects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category-selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception.