mlsanityData to 7 Oct 2026

Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge

arXiv 1609.06647 ↗First posted 21 Sep 2016cs.CVcited by 116

Not on the map. Why?

Abstract (arXiv)

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. Finally, given the recent surge of interest in this task, a competition was organized in 2015 using the newly released COCO dataset. We describe and analyze the various improvements we applied to our own baseline and show the resulting performance in the competition, which we won ex-aequo with a team from Microsoft Research, and provide an open source implementation in TensorFlow.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 34; 2022: 21; 2023: 31; 2024: 18; 2025: 9; 2026: 3.

116 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  6. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  7. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  8. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  9. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  10. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  11. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  12. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
38 more
  1. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  2. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  3. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 68
  4. 2024Common Sense Reasoning for Deepfake Detectioncited by 64
  5. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  6. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 60
  7. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  8. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  9. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 51
  10. 2021Visual Goal-Step Inference using wikiHowcited by 42
  11. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  12. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  13. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  14. 2023Retrieval-augmented Image Captioningcited by 28
  15. 2023A Review of Deep Learning for Video Captioningcited by 28
  16. 2021Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioningcited by 27
  17. 2023Towards Local Visual Modeling for Image Captioningcited by 26
  18. 2021MusCaps: Generating Captions for Music Audiocited by 23
  19. 2022Prompt-based Learning for Unpaired Image Captioningcited by 23
  20. 2021WikiGraphs: A Wikipedia Text - Knowledge Graph Paired Datasetcited by 20
  21. 2021Detector-Free Weakly Supervised Grounding by Separationcited by 16
  22. 2023Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignmentcited by 15
  23. 2021Caption Enriched Samples for Improving Hateful Memes Detectioncited by 14
  24. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14
  25. 2022Aligning Source Visual and Target Language Domains for Unpaired Video Captioningcited by 14
  26. 2023ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Datacited by 14
  27. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  28. 2022Symbolic music generation conditioned on continuous-valued emotionscited by 12
  29. 2024Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Trainingcited by 12
  30. 2024Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approachcited by 12
  31. 2024Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similaritycited by 12
  32. 2025Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perceptioncited by 12
  33. 2023Diagnosing Human-object Interaction Detectorscited by 11
  34. 2021Graphine: A Dataset for Graph-aware Terminology Definition Generationcited by 10
  35. 2023CgT-GAN: CLIP-guided Text GAN for Image Captioningcited by 10
  36. 2024Progress-Aware Video Frame Captioningcited by 10
  37. 2022Distance Matters in Human-Object Interaction Detectioncited by 8
  38. 2021Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scenecited by 7

50 most cited of 116 All 116 (JSON)