mlsanityData to 7 Oct 2026

Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning

arXiv 1612.01887 ↗First posted 6 Dec 2016cs.CV cs.AIcited by 231

Not on the map. Why?

Abstract (arXiv)

Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict non-visual words such as "the" and "of". Other words that may seem visual can often be predicted reliably just from the language model e.g., "sign" after "behind a red stop" or "phone" following "talking on a cell". In this paper, we propose a novel adaptive attention model with a visual sentinel. At each time step, our model decides whether to attend to the image (and if so, to which regions) or to the visual sentinel. The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation. We test our method on the COCO image captioning 2015 challenge dataset and Flickr30K. Our approach sets the new state-of-the-art by a significant margin.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 70; 2022: 55; 2023: 48; 2024: 32; 2025: 19; 2026: 6.

231 papers.

Sort:

By area:

  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  2. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  3. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  4. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  5. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  6. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  7. 2022Task Residual for Tuning Vision-Language Modelscited by 122
  8. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  9. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 111
  10. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  11. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  12. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 92
38 more
  1. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  2. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  3. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  4. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  5. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  6. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  7. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  8. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 71
  9. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  10. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  11. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  12. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 60
  13. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 58
  14. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  15. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  16. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  17. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  18. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  19. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 42
  20. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  21. 2023Changes to Captions: An Attentive Network for Remote Sensing Change Captioningcited by 41
  22. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  23. 2023ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoningcited by 39
  24. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  25. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  26. 2022Measuring Representational Harms in Image Captioningcited by 36
  27. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  28. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  29. 2021In Defense of Scene Graphs for Image Captioningcited by 31
  30. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  31. 2021Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioningcited by 27
  32. 2023Towards Local Visual Modeling for Image Captioningcited by 26
  33. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  34. 2021Automated Generation of Accurate & Fluent Medical X-ray Reportscited by 24
  35. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  36. 2021Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioningcited by 23
  37. 2022Prompt-based Learning for Unpaired Image Captioningcited by 23
  38. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23

50 most cited of 231 All 231 (JSON)