mlsanityData to 7 Oct 2026

What Makes for Good Image Captions?

arXiv 2405.00485 ↗First posted 1 May 2024cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper establishes a formal information-theoretic framework for image captioning, conceptualizing captions as compressed linguistic representations that selectively encode semantic units in images. Our framework posits that good image captions should balance three key aspects: informationally sufficient, minimally redundant, and readily comprehensible by humans. By formulating these aspects as quantitative measures with adjustable weights, our framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements. To demonstrate its applicability, we introduce the Pyramid of Captions (PoCa) method, which generates enriched captions by integrating local and global visual information. We present both theoretical proof that PoCa improves caption quality under certain assumptions, and empirical validation of its effectiveness across various image captioning models and datasets.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 45 of its 71 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

45 of 71 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
33 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  12. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  13. 2023Improving CLIP Training with Language Rewritescited by 216
  14. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  15. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  16. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  17. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  18. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  19. 2020Captioning Images Taken by People Who Are Blindcited by 108
  20. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  21. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  22. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  23. 2018Unsupervised Image Captioningcited by 79
  24. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  25. 2024When Do We Not Need Larger Vision Models?cited by 65
  26. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  27. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 54
  28. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  29. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  30. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignmentcited by 28
  31. 2017Recurrent Topic-Transition GAN for Visual Paragraph Generationcited by 25
  32. 2019Convolutional Auto-encoding of Sentence Topics for Image Paragraph Generationcited by 8
  33. 2023VLIS: Unimodal Language Models Guide Multimodal Language Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  2. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  3. 2025Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perceptioncited by 12
  4. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  5. 2025Fair Foundation Models for Medical Image Analysis: Challenges and Perspectivescited by 3
  6. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
  7. 2025Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captionscited by 1
  8. 2025Image Embedding Sampling Method for Diverse Captioningcited by 0
  9. 2026Evaluating Remote Sensing Image Captions Beyond Metric Biasescited by 0
  10. 2026BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understandingcited by 0
  11. 2026See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RLcited by 0