mlsanityData to 14 Oct 2026

From Captions to Visual Concepts and Back

arXiv 1411.4952 ↗First posted 18 Nov 2014cs.CV cs.CLcited by 109

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Nov 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 34; 2022: 26; 2023: 23; 2024: 9; 2025: 13; 2026: 3.

109 papers.

Sort:

By area:

  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  3. 2024Large Language Models: A Surveycited by 525
  4. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  5. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  6. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  7. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  8. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  9. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  10. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  11. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  12. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 86
38 more
  1. 2024Toward Visual Grounding: A Surveycited by 52
  2. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  3. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  4. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  5. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  6. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Modelscited by 36
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  8. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 34
  9. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  10. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 26
  11. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  12. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 25
  13. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  14. 2022Noise-aware Learning from Web-crawled Image-Text Data for Image Captioningcited by 22
  15. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentationcited by 21
  16. 2021Recursive Contour Saliency Blending Network for Accurate Salient Object Detectioncited by 20
  17. 2022SelfReformer: Self-Refined Network with Transformer for Salient Object Detectioncited by 20
  18. 2022Medical Image Captioning via Generative Pretrained Transformerscited by 20
  19. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  20. 2021Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videoscited by 18
  21. 2023Towards Diverse Binary Segmentation via A Simple yet General Gated Networkcited by 17
  22. 2023Improved Visual Grounding through Self-Consistent Explanationscited by 17
  23. 2021Detector-Free Weakly Supervised Grounding by Separationcited by 16
  24. 2021Contextualized Keyword Representations for Multi-modal Retinal Image Captioningcited by 16
  25. 2021Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"cited by 16
  26. 2023Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignmentcited by 15
  27. 2021Computer Vision and Conflicting Values: Describing People with Automated Alt Textcited by 14
  28. 2022Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrievalcited by 14
  29. 2022Aligning Source Visual and Target Language Domains for Unpaired Video Captioningcited by 14
  30. 2021ConRPG: Paraphrase Generation using Contexts as Regularizercited by 13
  31. 2021Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignmentcited by 13
  32. 2022CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrievalcited by 13
  33. 2021R3Net:Relation-embedded Representation Reconstruction Network for Change Captioningcited by 12
  34. 2022Joint Learning of Salient Object Detection, Depth Estimation and Contour Extractioncited by 12
  35. 2021Neural Attention for Image Captioning: Review of Outstanding Methodscited by 11
  36. 2022What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputscited by 11
  37. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 10
  38. 2023Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questionscited by 9

50 most cited of 109 All 109 (JSON)