# A Sensorimotor Vision Transformer

arXiv: https://arxiv.org/abs/2504.02536 · HTML: https://mlsanity.com/paper/2504.02536/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [On The Computational Complexity of Self-Attention](https://mlsanity.com/paper/2209.04881/) (2022)
- [TransMix: Attend to Mix for Vision Transformers](https://mlsanity.com/paper/2111.09833/) (2021)
- [A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond](https://mlsanity.com/paper/2208.00173/) (2022)
- [Saccader: Improving Accuracy of Hard Attention Models for Vision](https://mlsanity.com/paper/1908.07644/) (2019)
- [A survey of the Vision Transformers and their CNN-Transformer based Variants](https://mlsanity.com/paper/2305.09880/) (2023)
- [The Need for Speed: Pruning Transformers with One Recipe](https://mlsanity.com/paper/2403.17921/) (2024)
- [SALISA: Saliency-based Input Sampling for Efficient Video Object Detection](https://mlsanity.com/paper/2204.02397/) (2022)
- [SSBNet: Improving Visual Recognition Efficiency by Adaptive Sampling](https://mlsanity.com/paper/2207.11511/) (2022)
- [Modeling Human Eye Movements with Neural Networks in a Maze-Solving Task](https://mlsanity.com/paper/2212.10367/) (2022)

## Abstract (arXiv)

This paper presents the Sensorimotor Transformer (SMT), a vision model inspired by human saccadic eye movements that prioritize high-saliency regions in visual input to enhance computational efficiency and reduce memory consumption. Unlike traditional models that process all image patches uniformly, SMT identifies and selects the most salient patches based on intrinsic two-dimensional (i2D) features, such as corners and occlusions, which are known to convey high-information content and align with human fixation patterns. The SMT architecture uses this biological principle to leverage vision transformers to process only the most informative patches, allowing for a substantial reduction in memory usage that scales with the sequence length of selected patches. This approach aligns with visual neuroscience findings, suggesting that the human visual system optimizes information gathering through selective, spatially dynamic focus. Experimental evaluations on Imagenet-1k demonstrate that SMT achieves competitive top-1 accuracy while significantly reducing memory consumption and computational complexity, particularly when a limited number of patches is used. This work introduces a saccade-like selection mechanism into transformer-based vision models, offering an efficient alternative for image analysis and providing new insights into biologically motivated architectures for resource-constrained applications.