# Learning Correlation Structures for Vision Transformers

arXiv: https://arxiv.org/abs/2404.03924 · HTML: https://mlsanity.com/paper/2404.03924/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 92 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)

## Built on by (most cited first, 15 of 15)

- [Breaking the Low-Rank Dilemma of Linear Attention](https://mlsanity.com/paper/2411.07635/) (2024)
- [Rectifying Magnitude Neglect in Linear Attention](https://mlsanity.com/paper/2507.00698/) (2025)
- [DAMamba: Vision State Space Model with Dynamic Adaptive Scan](https://mlsanity.com/paper/2502.12627/) (2025)
- [Sparse Imagination for Efficient Visual World Model Planning](https://mlsanity.com/paper/2506.01392/) (2025)
- [Vision Transformer with Sparse Scan Prior](https://mlsanity.com/paper/2405.13335/) (2024)
- [Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens](https://mlsanity.com/paper/2405.13337/) (2024)
- [SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attention](https://mlsanity.com/paper/2509.12817/) (2025)
- [LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel](https://mlsanity.com/paper/2604.20368/) (2026)
- [Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention](https://mlsanity.com/paper/2506.21137/) (2025)
- [Exploring High-Order Self-Similarity for Video Understanding](https://mlsanity.com/paper/2604.20760/) (2026)
- [Tempered Self-Similarity Alignment for Physically Plausible Video Generation](https://mlsanity.com/paper/2605.24962/) (2026)
- [Dynamic Universal Approximation Theory: The Basic Theory for Deep Learning-Based Computer Vision Models](https://mlsanity.com/paper/2407.17480/) (2024)
- [Multi-modal and Multi-view Fundus Image Fusion for Retinopathy Diagnosis via Multi-scale Cross-attention and Shifted Window Self-attention](https://mlsanity.com/paper/2504.09106/) (2025)
- [Video Understanding by Design: How Datasets Shape Video Models](https://mlsanity.com/paper/2509.09151/) (2025)
- [MirrorLA: Reflecting Feature Map for Vision Linear Attention](https://mlsanity.com/paper/2602.04346/) (2026)

## Abstract (arXiv)

We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recognizing space-time structures of key-query correlations via convolution and uses them to dynamically aggregate local contexts of value features. This effectively leverages rich structural patterns in images and videos such as scene layouts, object motion, and inter-object relations. Using StructSA as a main building block, we develop the structural vision transformer (StructViT) and evaluate its effectiveness on both image and video classification tasks, achieving state-of-the-art results on ImageNet-1K, Kinetics-400, Something-Something V1 & V2, Diving-48, and FineGym.