mlsanityData to 7 Oct 2026

Attention on Attention for Image Captioning

arXiv 1908.06954 ↗First posted 19 Aug 2019cs.CVcited by 289

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.

Key facts. Cited by 289 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 82; 2023: 61; 2024: 32; 2025: 27; 2026: 5.

289 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  4. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  5. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  7. 2022Multimodal Learning with Transformers: A Surveycited by 399
  8. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  9. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  10. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  11. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  12. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
38 more
  1. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  2. 2022AiATrack: Attention in Attention for Transformer Visual Trackingcited by 137
  3. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  4. 2021Causal Attention for Vision-Language Taskscited by 111
  5. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  6. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  7. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  8. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  9. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  10. 2021Spatially-Adaptive Image Restoration using Distortion-Guided Networkscited by 77
  11. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  12. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  13. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  14. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  15. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 71
  16. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  17. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  18. 2021TransZero: Attribute-guided Transformer for Zero-Shot Learningcited by 53
  19. 2022NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Modelscited by 47
  20. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  21. 2024Context-Guided Spatio-Temporal Video Groundingcited by 45
  22. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  23. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  24. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  25. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41
  26. 2023Changes to Captions: An Attentive Network for Remote Sensing Change Captioningcited by 41
  27. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39
  28. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  29. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  30. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  31. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  32. 2022End-to-End Transformer Based Model for Image Captioningcited by 37
  33. 2022TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editingcited by 36
  34. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  35. 2023Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reportscited by 34
  36. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  37. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  38. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33

50 most cited of 289 All 289 (JSON)