mlsanityData to 14 Oct 2026

Learning Correlation Structures for Vision Transformers

arXiv 2404.03924 ↗First posted 5 Apr 2024cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recognizing space-time structures of key-query correlations via convolution and uses them to dynamically aggregate local contexts of value features. This effectively leverages rich structural patterns in images and videos such as scene layouts, object motion, and inter-object relations. Using StructSA as a main building block, we develop the structural vision transformer (StructViT) and evaluate its effectiveness on both image and video classification tasks, achieving state-of-the-art results on ImageNet-1K, Kinetics-400, Something-Something V1 & V2, Diving-48, and FineGym.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 80 of its 92 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

80 of 92 references.

References · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2017Mask R-CNNcited by 7,277
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
38 more
  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2022A ConvNet for the 2020scited by 3,579
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  6. 2017Non-local Neural Networkscited by 2,293
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2018SlowFast Networks for Video Recognitioncited by 1,867
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  12. 2021ViViT: A Video Vision Transformercited by 1,620
  13. 2017Random Erasing Data Augmentationcited by 1,239
  14. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  15. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  16. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  17. 2021Video Swin Transformercited by 1,091
  18. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  19. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  20. 2019When Does Label Smoothing Help?cited by 988
  21. 2016Deep Networks with Stochastic Depthcited by 979
  22. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  23. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  24. 2021Multiscale Vision Transformerscited by 897
  25. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  26. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  27. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  28. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  29. 2019Panoptic Feature Pyramid Networkscited by 645
  30. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  31. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  32. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  33. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  34. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  35. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  36. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  37. 2017Temporal Relational Reasoning in Videoscited by 324
  38. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  2. 2025Rectifying Magnitude Neglect in Linear Attentioncited by 16
  3. 2025DAMamba: Vision State Space Model with Dynamic Adaptive Scancited by 9
  4. 2025Sparse Imagination for Efficient Visual World Model Planningcited by 7
  5. 2024Vision Transformer with Sparse Scan Priorcited by 6
  6. 2024Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokenscited by 5
  7. 2025SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attentioncited by 3
  8. 2026LaplacianFormer:Rethinking Linear Attention with Laplacian Kernelcited by 3
  9. 2025Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attentioncited by 2
  10. 2026Exploring High-Order Self-Similarity for Video Understandingcited by 1
  11. 2026Tempered Self-Similarity Alignment for Physically Plausible Video Generationcited by 1
  12. 2024Dynamic Universal Approximation Theory: The Basic Theory for Deep Learning-Based Computer Vision Modelscited by 0
3 more
  1. 2025Multi-modal and Multi-view Fundus Image Fusion for Retinopathy Diagnosis via Multi-scale Cross-attention and Shifted Window Self-attentioncited by 0
  2. 2025Video Understanding by Design: How Datasets Shape Video Modelscited by 0
  3. 2026MirrorLA: Reflecting Feature Map for Vision Linear Attentioncited by 0