# Scratching Visual Transformer's Back with Uniform Attention

arXiv: https://arxiv.org/abs/2210.08457 · HTML: https://mlsanity.com/paper/2210.08457/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 71 references that are in the map.
- First posted on arXiv 16 Oct 2022.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)

## Built on by (most cited first, 16 of 16)

- [Scene Adaptive Sparse Transformer for Event-based Object Detection](https://mlsanity.com/paper/2404.01882/) (2024)
- [WorldAfford: Affordance Grounding based on Natural Language Instructions](https://mlsanity.com/paper/2405.12461/) (2024)
- [Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2405.15169/) (2024)
- [Multi-view Pyramid Transformer: Look Coarser to See Broader](https://mlsanity.com/paper/2512.07806/) (2025)
- [Exploring Consistency in Cross-Domain Transformer for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2211.14703/) (2022)
- [A Manifold Representation of the Key in Vision Transformers](https://mlsanity.com/paper/2402.00534/) (2024)
- [UnitNorm: Rethinking Normalization for Transformers in Time Series](https://mlsanity.com/paper/2405.15903/) (2024)
- [Relieving the Over-Aggregating Effect in Graph Transformers](https://mlsanity.com/paper/2510.21267/) (2025)
- [Rethinking Spatio-Temporal Transformer for Traffic Prediction:Multi-level Multi-view Augmented Learning Framework](https://mlsanity.com/paper/2406.11921/) (2024)
- [MxT: Mamba x Transformer for Image Inpainting](https://mlsanity.com/paper/2407.16126/) (2024)
- [Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention](https://mlsanity.com/paper/2507.19891/) (2025)
- [Do We Really Need Adaptive Global Spatial Attention for Traffic Forecasting?](https://mlsanity.com/paper/2607.12462/) (2026)
- [Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention](https://mlsanity.com/paper/2512.04551/) (2025)
- [Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs](https://mlsanity.com/paper/2603.14892/) (2026)
- [DnA: Denoising Attention for Visual Tasks](https://mlsanity.com/paper/2606.27372/) (2026)
- [QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices](https://mlsanity.com/paper/2609.30592/) (2026)

## Abstract (arXiv)

The favorable performance of Vision Transformers (ViTs) is often attributed to the multi-head self-attention (MSA). The MSA enables global interactions at each layer of a ViT model, which is a contrasting feature against Convolutional Neural Networks (CNNs) that gradually increase the range of interaction across multiple layers. We study the role of the density of the attention. Our preliminary analyses suggest that the spatial interactions of attention maps are close to dense interactions rather than sparse ones. This is a curious phenomenon, as dense attention maps are harder for the model to learn due to steeper softmax gradients around them. We interpret this as a strong preference for ViT models to include dense interaction. We thus manually insert the uniform attention to each layer of ViT models to supply the much needed dense interactions. We call this method Context Broadcasting, CB. We observe that the inclusion of CB reduces the degree of density in the original attention maps and increases both the capacity and generalizability of the ViT models. CB incurs negligible costs: 1 line in your model code, no additional parameters, and minimal extra operations.