# FoveaTer: Foveated Transformer for Image Classification

arXiv: https://arxiv.org/abs/2105.14173 · HTML: https://mlsanity.com/paper/2105.14173/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 29 May 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Technical Report on the CleverHans v2.1.0 Adversarial Examples Library](https://mlsanity.com/paper/1610.00768/) (2016)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency](https://mlsanity.com/paper/2006.16736/) (2020)
- [Mind the Pad -- CNNs can Develop Blind Spots](https://mlsanity.com/paper/2010.02178/) (2020)
- [Foveation-based Mechanisms Alleviate Adversarial Examples](https://mlsanity.com/paper/1511.06292/) (2015)
- [Biologically Inspired Mechanisms for Adversarial Robustness](https://mlsanity.com/paper/2006.16427/) (2020)
- [Emergent Properties of Foveated Perceptual Systems](https://mlsanity.com/paper/2006.07991/) (2020)
- [Object Detection Through Exploration With A Foveated Visual Field](https://mlsanity.com/paper/1408.0814/) (2014)
- [Towards Metamerism via Foveated Style Transfer](https://mlsanity.com/paper/1705.10041/) (2017)
- [Predicting Goal-directed Attention Control Using Inverse-Reinforcement Learning](https://mlsanity.com/paper/2001.11921/) (2020)
- [Recurrent Attention Model with Log-Polar Mapping is Robust against Adversarial Attacks](https://mlsanity.com/paper/2002.05388/) (2020)

## Built on by (most cited first, 28 of 28)

- [Scaled ReLU Matters for Training Vision Transformers](https://mlsanity.com/paper/2109.03810/) (2021)
- [PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution](https://mlsanity.com/paper/2403.07589/) (2024)
- [Peripheral Vision Transformer](https://mlsanity.com/paper/2206.06801/) (2022)
- [Vision Transformer Adapters for Generalizable Multitask Learning](https://mlsanity.com/paper/2308.12372/) (2023)
- [On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation](https://mlsanity.com/paper/2112.07173/) (2021)
- [A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains](https://mlsanity.com/paper/2310.13849/) (2023)
- [Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks](https://mlsanity.com/paper/2202.00838/) (2022)
- [Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://mlsanity.com/paper/2507.15833/) (2025)
- [Foveation in the Era of Deep Learning](https://mlsanity.com/paper/2312.01450/) (2023)
- [FBS: Modeling Native Parallel Reading inside a Transformer](https://mlsanity.com/paper/2601.21708/) (2026)
- [Training on Foveated Images Improves Robustness to Adversarial Attacks](https://mlsanity.com/paper/2308.00854/) (2023)
- [Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation](https://mlsanity.com/paper/2506.11131/) (2025)
- [Human Eyes Inspired Recurrent Neural Networks are More Robust Against Adversarial Noises](https://mlsanity.com/paper/2206.07282/) (2022)
- [Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation](https://mlsanity.com/paper/2603.23491/) (2026)
- [Vision encoders should be image size agnostic and task driven](https://mlsanity.com/paper/2508.16317/) (2025)
- [Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models](https://mlsanity.com/paper/2604.21079/) (2026)
- [Learning to search for and detect objects in foveal images using deep learning](https://mlsanity.com/paper/2304.05741/) (2023)
- [FOVI: A biologically-inspired foveated interface for deep vision models](https://mlsanity.com/paper/2602.03766/) (2026)
- [URLOST: Unsupervised Representation Learning without Stationarity or Topology](https://mlsanity.com/paper/2310.04496/) (2023)
- [On Inherent Adversarial Robustness of Active Vision Systems](https://mlsanity.com/paper/2404.00185/) (2024)
- [FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2408.02123/) (2024)
- [Fixed Inter-Neuron Covariability Induces Adversarial Robustness](https://mlsanity.com/paper/2308.03956/) (2023)
- [Self-supervised pretraining for an iterative image size agnostic vision transformer](https://mlsanity.com/paper/2604.20392/) (2026)
- [Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers](https://mlsanity.com/paper/2607.09480/) (2026)
- [HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams](https://mlsanity.com/paper/2608.02140/) (2026)
- [Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans](https://mlsanity.com/paper/2608.16514/) (2026)
- [Active Spiking Perception: The Membrane Potential as a Belief State for Anytime 3D Point Cloud Recognition](https://mlsanity.com/paper/2608.19232/) (2026)
- [Adaptive Observation Control for Active Visual Recognition](https://mlsanity.com/paper/2610.19624/) (2026)

## Abstract (arXiv)

Many animals and humans process the visual field with a varying spatial resolution (foveated vision) and use peripheral processing to make eye movements and point the fovea to acquire high-resolution information about objects of interest. This architecture results in computationally efficient rapid scene exploration. Recent progress in self-attention-based Vision Transformers, an alternative to the traditionally convolution-reliant computer vision systems. However, the Transformer models do not explicitly model the foveated properties of the visual system nor the interaction between eye movements and the classification task. We propose Foveated Transformer (FoveaTer) model, which uses pooling regions and eye movements to perform object classification tasks using a Vision Transformer architecture. Using square pooling regions or biologically-inspired radial-polar pooling regions, our proposed model pools the image features from the convolution backbone and uses the pooled features as an input to transformer layers. It decides on subsequent fixation location based on the attention assigned by the Transformer to various locations from past and present fixations. It dynamically allocates more fixation/computational resources to more challenging images before making the final image category decision. Using five ablation studies, we evaluate the contribution of different components of the Foveated model. We perform a psychophysics scene categorization task and use the experimental data to find a suitable radial-polar pooling region combination. We also show that the Foveated model better explains the human decisions in a scene categorization task than a Baseline model. We demonstrate our model's robustness against PGD adversarial attacks with both types of pooling regions, where we see the Foveated model outperform the Baseline model.