mlsanityData to 7 Oct 2026

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

arXiv 1612.03928 ↗First posted 12 Dec 2016cs.CVcited by 1,059

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer

Key facts. Cited by 1,059 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 219; 2022: 219; 2023: 188; 2024: 189; 2025: 117; 2026: 127.

1,059 papers.

Sort:

By area:

  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2024Depth Anything V2cited by 1,385
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  5. 2022Decoupled Knowledge Distillationcited by 380
  6. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  7. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  8. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  9. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  10. 2022Knowledge Distillation from A Stronger Teachercited by 204
  11. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  12. 2021Does Knowledge Distillation Really Work?cited by 187
38 more
  1. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  2. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  3. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  4. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  5. 2021Focal and Global Knowledge Distillation for Detectorscited by 138
  6. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  7. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  8. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  9. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  10. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  11. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 118
  12. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  13. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  14. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  15. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  16. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  17. 2022Masked Generative Distillationcited by 107
  18. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  19. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  20. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  21. 2024Logit Standardization in Knowledge Distillationcited by 101
  22. 2023Diffusion Model as Representation Learnercited by 95
  23. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 93
  24. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  25. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  26. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  27. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  28. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  29. 2023SCPNet: Semantic Scene Completion on Point Cloudcited by 80
  30. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
  31. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  32. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  33. 2024D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinementcited by 73
  34. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  35. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  36. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  37. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
  38. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69

50 most cited of 1,059 All 1,059 (JSON)