mlsanityData to 14 Oct 2026

Multi-Head Attention: Collaborate Instead of Concatenate

arXiv 2006.16362 ↗First posted 29 Jun 2020cs.LG cs.CL stat.MLcited by 60

Not on the map. Why?

Abstract (arXiv)

Attention layers are widely used in natural language processing (NLP) and are beginning to influence computer vision architectures. Training very large transformer models allowed significant improvement in both fields, but once trained, these networks show symptoms of over-parameterization. For instance, it is known that many attention heads can be pruned without impacting accuracy. This work aims to enhance current understanding on how multiple heads interact. Motivated by the observation that attention heads learn redundant key/query projections, we propose a collaborative multi-head attention layer that enables heads to learn shared projections. Our scheme decreases the number of parameters in an attention layer and can be used as a drop-in replacement in any transformer architecture. Our experiments confirm that sharing key/query dimensions can be exploited in language understanding, machine translation and vision. We also show that it is possible to re-parametrize a pre-trained multi-head attention layer into our collaborative attention layer. Collaborative multi-head attention reduces the size of the key and query projections by 4 for same accuracy and speed. Our code is public.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 7; 2024: 8; 2025: 16; 2026: 13.

60 papers.

Sort:

By area:

  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  2. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  3. 2021A Survey of Transformerscited by 401
  4. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  5. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  6. 2023Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEscited by 68
  7. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  8. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 44
  9. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  10. 2025DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformerscited by 17
  11. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  12. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
38 more
  1. 2023Accurate Retraining-free Pruning for Pretrained Encoder-based Language Modelscited by 11
  2. 2025SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimizationcited by 11
  3. 2024Improving Transformers with Dynamically Composable Multi-Head Attentioncited by 10
  4. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  5. 2024GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projectioncited by 8
  6. 2025MAVEN: Multi-modal Attention for Valence-Arousal Emotion Networkcited by 8
  7. 2021Local Multi-Head Channel Self-Attention for Facial Expression Recognitioncited by 7
  8. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  9. 2022Rethinking Attention Mechanism in Time Series Classificationcited by 5
  10. 2024Latent Diffusion for Medical Image Segmentation: End to end learning for fast sampling and accuracycited by 5
  11. 2022Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysiscited by 4
  12. 2022Group Generalized Mean Pooling for Vision Transformercited by 4
  13. 2025The geometry of BERTcited by 4
  14. 2025TacEleven: generative tactic discovery for football open playcited by 4
  15. 2026Tucker Attention: A generalization of approximate attention mechanismscited by 3
  16. 2021Multi-Image Visual Question Answeringcited by 2
  17. 2022Adaptive Multi-Neighborhood Attention based Transformer for Graph Representation Learningcited by 2
  18. 2024Generalized Probabilistic Attention Mechanism in Transformerscited by 2
  19. 2025Untangling Component Imbalance in Hybrid Linear Attention Conversion Methodscited by 2
  20. 2025The SAM2-to-SAM3 Gap in the Segment Anything Model Family: Why Prompt-Based Expertise Fails in Concept-Driven Image Segmentationcited by 2
  21. 2023Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Modelscited by 1
  22. 2023Finding the Pillars of Strength for Multi-Head Attentioncited by 1
  23. 2025Knocking-Heads Attentioncited by 1
  24. 2026Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledgecited by 1
  25. 2021MultiHead MultiModal Deep Interest Recommendation Networkcited by 0
  26. 2022Performance Comparison of Simple Transformer and Res-CNN-BiLSTM for Cyberbullying Classificationcited by 0
  27. 2022Localizing Anatomical Landmarks in Ocular Images using Zoom-In Attentive Networkscited by 0
  28. 2023Short-Term Load Forecasting Using A Particle-Swarm Optimized Multi-Head Attention-Augmented CNN-LSTM Networkcited by 0
  29. 2023MUSTANG: Multi-Stain Self-Attention Graph Multiple Instance Learning Pipeline for Histopathology Whole Slide Imagescited by 0
  30. 2024ART: Artifact Removal Transformer for Reconstructing Noise-Free Multichannel Electroencephalographic Signalscited by 0
  31. 2025Separated Inter/Intra-Modal Fusion Prompts for Compositional Zero-Shot Learningcited by 0
  32. 2025Converting Transformers into DGNNs Formcited by 0
  33. 2025Selective Complementary Feature Fusion and Modal Feature Compression Interaction for Brain Tumor Segmentationcited by 0
  34. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Frameworkcited by 0
  35. 2025Heterogeneous network drug-target interaction prediction model based on graph wavelet transform and multi-level contrastive learningcited by 0
  36. 2025Graph Collaborative Attention Network for Link Prediction in Knowledge Graphscited by 0
  37. 2025Denoising-While-Completing Network (DWCNet): Robust Point Cloud Completion Under Corruptioncited by 0
  38. 2025Load Forecasting on A Highly Sparse Electrical Load Dataset Using Gaussian Interpolationcited by 0

50 most cited of 60 All 60 (JSON)