# TransNeXt: Robust Foveal Visual Perception for Vision Transformers

arXiv: https://arxiv.org/abs/2311.17132 · HTML: https://mlsanity.com/paper/2311.17132/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 82 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 62)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)

## Built on by (most cited first, 30 of 61)

- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition](https://mlsanity.com/paper/2403.17695/) (2024)
- [MoH: Multi-Head Attention as Mixture-of-Head Attention](https://mlsanity.com/paper/2410.11842/) (2024)
- [SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation](https://mlsanity.com/paper/2412.11890/) (2024)
- [Learned Image Compression with Dictionary-based Entropy Model](https://mlsanity.com/paper/2504.00496/) (2025)
- [VSSD: Vision Mamba with Non-Causal State Space Duality](https://mlsanity.com/paper/2407.18559/) (2024)
- [ViT3: Unlocking Test-Time Training in Vision](https://mlsanity.com/paper/2512.01643/) (2025)
- [Hypergraph Vision Transformers: Images are More than Nodes, More than Edges](https://mlsanity.com/paper/2504.08710/) (2025)
- [PVChat: Personalized Video Chat with One-Shot Learning](https://mlsanity.com/paper/2503.17069/) (2025)
- [Cross-Layer Feature Pyramid Transformer for Small Object Detection in Aerial Images](https://mlsanity.com/paper/2407.19696/) (2024)
- [Frequency-Dynamic Attention Modulation for Dense Prediction](https://mlsanity.com/paper/2507.12006/) (2025)
- [iFormer: Integrating ConvNet and Transformer for Mobile Application](https://mlsanity.com/paper/2501.15369/) (2025)
- [DAMamba: Vision State Space Model with Dynamic Adaptive Scan](https://mlsanity.com/paper/2502.12627/) (2025)
- [Unleashing the Power of Generic Segmentation Models: A Simple Baseline for Infrared Small Target Detection](https://mlsanity.com/paper/2409.04714/) (2024)
- [Filling Missing Values Matters for Range Image-Based Point Cloud Segmentation](https://mlsanity.com/paper/2405.10175/) (2024)
- [A Composite Predictive-Generative Approach to Monaural Universal Speech Enhancement](https://mlsanity.com/paper/2505.24576/) (2025)
- [PhyG-MoE: A Physics-Guided Mixture-of-Experts Framework for Energy-Efficient GNSS Interference Recognition](https://mlsanity.com/paper/2601.12798/) (2026)
- [HGFormer: Topology-Aware Vision Transformer with HyperGraph Learning](https://mlsanity.com/paper/2504.02440/) (2025)
- [S2AFormer: Strip Self-Attention for Efficient Vision Transformer](https://mlsanity.com/paper/2505.22195/) (2025)
- [A Manifold Representation of the Key in Vision Transformers](https://mlsanity.com/paper/2402.00534/) (2024)
- [MambaMIC: An Efficient Baseline for Microscopic Image Classification with State Space Models](https://mlsanity.com/paper/2409.07896/) (2024)
- [SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers](https://mlsanity.com/paper/2409.19850/) (2024)
- [Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving](https://mlsanity.com/paper/2501.08083/) (2025)
- [Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation](https://mlsanity.com/paper/2502.02763/) (2025)
- [WoundAmbit: Bridging State-of-the-Art Semantic Segmentation and Real-World Wound Care](https://mlsanity.com/paper/2504.06185/) (2025)
- [Deformba: Vision State Space Model with Adaptive State Fusion](https://mlsanity.com/paper/2605.21308/) (2026)
- [DSDFormer: An Innovative Transformer-Mamba Framework for Robust High-Precision Driver Distraction Identification](https://mlsanity.com/paper/2409.05587/) (2024)
- [Low-Biased General Annotated Dataset Generation](https://mlsanity.com/paper/2412.10831/) (2024)
- [Edge Attention Module for Object Classification](https://mlsanity.com/paper/2502.03103/) (2025)
- [Cross Paradigm Representation and Alignment Transformer for Image Deraining](https://mlsanity.com/paper/2504.16455/) (2025)

## Abstract (arXiv)

Due to the depth degradation effect in residual connections, many efficient Vision Transformers models that rely on stacking layers for information exchange often fail to form sufficient information mixing, leading to unnatural visual perception. To address this issue, in this paper, we propose Aggregated Attention, a biomimetic design-based token mixer that simulates biological foveal vision and continuous eye movement while enabling each token on the feature map to have a global perception. Furthermore, we incorporate learnable tokens that interact with conventional queries and keys, which further diversifies the generation of affinity matrices beyond merely relying on the similarity between queries and keys. Our approach does not rely on stacking for information exchange, thus effectively avoiding depth degradation and achieving natural visual perception. Additionally, we propose Convolutional GLU, a channel mixer that bridges the gap between GLU and SE mechanism, which empowers each token to have channel attention based on its nearest neighbor image features, enhancing local modeling capability and model robustness. We combine aggregated attention and convolutional GLU to create a new visual backbone called TransNeXt. Extensive experiments demonstrate that our TransNeXt achieves state-of-the-art performance across multiple model sizes. At a resolution of 2242, TransNeXt-Tiny attains an ImageNet accuracy of 84.0%, surpassing ConvNeXt-B with 69% fewer parameters. Our TransNeXt-Base achieves an ImageNet accuracy of 86.2% and an ImageNet-A accuracy of 61.6% at a resolution of 3842, a COCO object detection mAP of 57.1, and an ADE20K semantic segmentation mIoU of 54.7.