mlsanityData to 14 Oct 2026

Image Captioning with Semantic Attention

arXiv 1603.03925 ↗First posted 12 Mar 2016cs.CVcited by 260

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics.

Key facts. Cited by 260 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 85; 2022: 66; 2023: 40; 2024: 34; 2025: 30; 2026: 5.

260 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  5. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  6. 2023Multimodal Large Language Models: A Surveycited by 195
  7. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  8. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  9. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 140
  10. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
38 more
  1. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  2. 2021Causal Attention for Vision-Language Taskscited by 111
  3. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  4. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  5. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 101
  6. 2022TableFormer: Table Structure Understanding with Transformerscited by 94
  7. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  8. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  9. 2021HDMI: High-order Deep Multiplex Infomaxcited by 82
  10. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  11. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  12. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 70
  13. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  14. 2021A Comprehensive Survey on Community Detection with Deep Learningcited by 63
  15. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 59
  16. 2022Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Groundingcited by 53
  17. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  18. 2021The Who in XAI: How AI Background Shapes Perceptions of AI Explanationscited by 48
  19. 2024Context-Guided Spatio-Temporal Video Groundingcited by 46
  20. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  21. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 44
  22. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  23. 2021Visual Navigation with Spatial Attentioncited by 43
  24. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  25. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  26. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  27. 2022Controllable Text Generation with Neurally-Decomposed Oraclecited by 41
  28. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  29. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  30. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  31. 2022Quantifying Societal Bias Amplification in Image Captioningcited by 37
  32. 2022Measuring Representational Harms in Image Captioningcited by 36
  33. 2021Semantic Grouping Network for Video Captioningcited by 34
  34. 2022Invariant Feature Learning for Generalized Long-Tailed Classificationcited by 34
  35. 2022M^4I: Multi-modal Models Membership Inferencecited by 34
  36. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  37. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  38. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 33

50 most cited of 260 All 260 (JSON)