mlsanityData to 7 Oct 2026

Hydra Attention: Efficient Attention with Many Heads

arXiv 2209.07484 ↗First posted 15 Sep 2022cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

While transformers have begun to dominate many tasks in vision, applying them to large images is still computationally difficult. A large reason for this is that self-attention scales quadratically with the number of tokens, which in turn, scales quadratically with the image size. On larger images (e.g., 1080p), over 60% of the total computation in the network is spent solely on creating and applying attention matrices. We take a step toward solving this issue by introducing Hydra Attention, an extremely efficient attention operation for Vision Transformers (ViTs). Paradoxically, this efficiency comes from taking multi-head attention to its extreme: by using as many attention heads as there are features, Hydra Attention is computationally linear in both tokens and features with no hidden constants, making it significantly faster than standard self-attention in an off-the-shelf ViT-B/16 by a factor of the token count. Moreover, Hydra Attention retains high accuracy on ImageNet and, in some cases, actually improves it.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

33 of 37 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2017Mask R-CNNcited by 7,259
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2020Rethinking Attention with Performerscited by 1,584
21 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  3. 2021Scaling Vision Transformerscited by 993
  4. 2021Multiscale Vision Transformerscited by 896
  5. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2020Quantifying Attention Flow in Transformerscited by 759
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  10. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  11. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  12. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  13. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  14. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  15. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  16. 2021Random Feature Attentioncited by 350
  17. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  18. 2021An Attention Free Transformercited by 111
  19. 2021Lite Vision Transformer with Enhanced Self-Attentioncited by 64
  20. 2021UFO-ViT: High Performance Linear Vision Transformer without Softmaxcited by 11
  21. 2022ViT-P: Rethinking Data-efficient Vision Transformers from Localitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 15; 2024: 20; 2025: 9; 2026: 6.

55 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  3. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  4. 2023Spike-driven Transformercited by 141
  5. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  6. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  7. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  8. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  9. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  10. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  11. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  12. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
38 more
  1. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  2. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  3. 2023PolySketchFormer: Fast Transformers via Sketching Polynomial Kernelscited by 33
  4. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  5. 2025PolaFormer: Polarity-aware Linear Attention for Vision Transformerscited by 30
  6. 2022Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inferencecited by 28
  7. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  8. 2022RGB no more: Minimally-decoded JPEG Vision Transformerscited by 18
  9. 2023Window Attention is Bugged: How not to Interpolate Position Embeddingscited by 18
  10. 2023DuETT: Dual Event Time Transformer for Electronic Health Recordscited by 17
  11. 2024SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalizationcited by 17
  12. 2025Rectifying Magnitude Neglect in Linear Attentioncited by 16
  13. 2023ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformercited by 15
  14. 2023To Transformers and Beyond: Large Language Models for the Genomecited by 15
  15. 2024CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applicationscited by 12
  16. 2024Sub-Adjacent Transformer: Improving Time Series Anomaly Detection with Reconstruction Error from Sub-Adjacent Neighborhoodscited by 10
  17. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  18. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 9
  19. 2024Optimizing the Deployment of Tiny Transformers on Low-Power MCUscited by 8
  20. 2023BinaryViT: Towards Efficient and Accurate Binary Vision Transformerscited by 7
  21. 2024LowFormer: Hardware Efficient Design for Convolutional Transformer Backbonescited by 6
  22. 2024PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformercited by 5
  23. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  24. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  25. 2026ViT-AdaLA: Adapting Vision Transformers with Linear Attentioncited by 4
  26. 2025Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attentioncited by 3
  27. 2024Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformerscited by 2
  28. 2024Discrete Cosine Transform Based Decorrelated Attention for Vision Transformerscited by 2
  29. 2025SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networkscited by 2
  30. 2025Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attentioncited by 2
  31. 2023Efficient Large-Scale Visual Representation Learning And Evaluationcited by 1
  32. 2023DualToken-ViT: Position-aware Efficient Vision Transformer with Dual Token Fusioncited by 1
  33. 2023Predicting Token Impact Towards Efficient Vision Transformercited by 0
  34. 2024Cascaded Multi-Scale Attention for Enhanced Multi-Scale Feature Extraction and Interaction with Low-Resolution Imagescited by 0
  35. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognitioncited by 0
  36. 2025MSLAU-Net: A Hybrid CNN-Transformer Network for Medical Image Segmentationcited by 0
  37. 2025DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUscited by 0
  38. 2025Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiencycited by 0

50 most cited of 55 All 55 (JSON)