mlsanityData to 14 Oct 2026

Talking-Heads Attention

arXiv 2003.02436 ↗First posted 5 Mar 2020cs.LG cs.NE cs.SDcited by 62

Not on the map. Why?

Abstract (arXiv)

We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation.While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 12; 2024: 8; 2025: 7; 2026: 20.

62 papers.

Sort:

By area:

  1. 2021Going deeper with Image Transformerscited by 764
  2. 2021A Survey of Transformerscited by 401
  3. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  4. 2022Transformers in Medical Imaging: A Surveycited by 245
  5. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 234
  6. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  7. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  8. 2021Vision Transformers with Patch Diversificationcited by 45
  9. 2021Refiner: Refining Self-attention for Vision Transformerscited by 38
  10. 2021MedGPT: Medical Concept Prediction from Clinical Narrativescited by 24
  11. 2023TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphscited by 16
  12. 2023Multi-Head State Space Model for Speech Recognitioncited by 14
38 more
  1. 2024Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasetscited by 14
  2. 2021WeChat Neural Machine Translation Systems for WMT21cited by 11
  3. 2023ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devicescited by 10
  4. 2024Improving Transformers with Dynamically Composable Multi-Head Attentioncited by 10
  5. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  6. 2025Multi-Token Attentioncited by 10
  7. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  8. 2024Enhancing Automatic Modulation Recognition through Robust Global Feature Extractioncited by 7
  9. 2025Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-offcited by 6
  10. 2021Streaming Transformer Transducer Based Speech Recognition Using Non-Causal Convolutioncited by 5
  11. 2024MultiMax: Sparse and Multi-Modal Attention Learningcited by 5
  12. 2025Devil is in the Uniformity: Exploring Diverse Learners within Transformer for Image Restorationcited by 5
  13. 2023How Much Context Does My Attention-Based ASR System Need?cited by 3
  14. 2026What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMscited by 3
  15. 2022FL-Tuning: Layer Tuning for Feed-Forward Network in Transformercited by 2
  16. 2023Entropic Score metric: Decoupling Topology and Size in Training-free NAScited by 2
  17. 2026From Mice to Trains: Amortized Bayesian Inference on Graph Datacited by 2
  18. 2026Explicit Multi-head Attention for Inter-head Interaction in Large Language Modelscited by 2
  19. 2026LoSA-Net: A Localized and Scale-Adaptive Network for Boundary-Sensitive Prediction of Perineural Invasion in 3D MRIcited by 2
  20. 2022BJTU-WeChat's Systems for the WMT22 Chat Translation Taskcited by 1
  21. 2022EIT: Enhanced Interactive Transformercited by 1
  22. 2023Finding the Pillars of Strength for Multi-Head Attentioncited by 1
  23. 2023MABViT -- Modified Attention Block Enhances Vision Transformerscited by 1
  24. 2024Beyond Isolated Heads: Multi-Overlapped-Head Self-Attention for Vision Transformerscited by 1
  25. 2024Dockformer: A transformer-based molecular docking paradigm for large-scale virtual screeningcited by 1
  26. 2025Knocking-Heads Attentioncited by 1
  27. 2026Softmax Linear Attention: Reclaiming Global Competitioncited by 1
  28. 2026Interleaved Head Attentioncited by 1
  29. 2026Invertible Query-Key Coupling Composes with Attention Mechanismscited by 1
  30. 2026TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Modelscited by 1
  31. 2026Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inferencecited by 1
  32. 2022FF2: A Feature Fusion Two-Stream Framework for Punctuation Restorationcited by 0
  33. 2023Semantic Feature Integration network for Fine-grained Visual Classificationcited by 0
  34. 2023Finding Stakeholder-Material Information from 10-K Reports using Fine-Tuned BERT and LSTM Modelscited by 0
  35. 2023ETDPC: A Multimodality Framework for Classifying Pages in Electronic Theses and Dissertationscited by 0
  36. 2023Memory-efficient Stochastic methods for Memory-based Transformerscited by 0
  37. 2024GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasetscited by 0
  38. 2025SAGE-Amine: Generative Amine Design with Multi-Property Optimization for Efficient CO2 Capturecited by 0

50 most cited of 62 All 62 (JSON)