# Beyond Isolated Heads: Multi-Overlapped-Head Self-Attention for Vision Transformers

arXiv: https://arxiv.org/abs/2410.14874 · HTML: https://mlsanity.com/paper/2410.14874/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Separable Self-attention for Mobile Vision Transformers](https://mlsanity.com/paper/2206.02680/) (2022)
- [Talking-Heads Attention](https://mlsanity.com/paper/2003.02436/) (2020)
- [MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features](https://mlsanity.com/paper/2209.15159/) (2022)
- [MS-DETR: Efficient DETR Training with Mixed Supervision](https://mlsanity.com/paper/2401.03989/) (2024)
- [Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.12333/) (2022)
- [Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets](https://mlsanity.com/paper/2407.19394/) (2024)
- [Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergence](https://mlsanity.com/paper/2112.13310/) (2021)
- [ACC-ViT : Atrous Convolution's Comeback in Vision Transformers](https://mlsanity.com/paper/2403.04200/) (2024)
- [Multi-Layer Dense Attention Decoder for Polyp Segmentation](https://mlsanity.com/paper/2403.18180/) (2024)

## Built on by (most cited first, 1 of 1)

- [PF3Det: A Prompted Foundation Feature Assisted Visual LiDAR 3D Detector](https://mlsanity.com/paper/2504.03563/) (2025)

## Abstract (arXiv)

Multi-Head Self-Attention (MHSA) is the cornerstone of Vision Transformers, allowing models to capture diverse feature representations by projecting tokens into independent subspaces. However, the standard MHSA strictly isolates these heads, preventing any information exchange during the attention computation itself. In this paper, we propose Multi-Overlapped-Head Self-Attention (MOHSA), a novel mechanism that replaces the hard division of attention heads with a soft, overlapping division. By allowing queries, keys, and values to partially overlap with those of adjacent heads, MOHSA fosters rich inter-head communication directly within the attention mechanism. We extensively investigate various overlap dimension scheduling strategies to identify optimal configurations. Comprehensive experiments in several Transformer architectures demonstrate that MOHSA outperforms standard MHSA on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1k datasets, offering a significant performance boost with negligible computational overhead. Code link: https://github.com/ZTX-100/MOHSA.