mlsanityData to 7 Oct 2026

ImageInWords: Unlocking Hyper-Detailed Image Descriptions

arXiv 2405.02793 ↗First posted 5 May 2024cs.CV cs.CLcited by 56

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the longstanding adage "an image is worth a thousand words," generating accurate hyper-detailed image descriptions remains unsolved. Trained on short web-scraped image text, vision-language models often generate incomplete descriptions with visual inconsistencies. We address this via a novel data-centric approach with ImageInWords (IIW), a carefully designed human-in-the-loop framework for curating hyper-detailed image descriptions. Human evaluations on IIW data show major gains compared to recent datasets (+66%) and GPT4V (+48%) across comprehensiveness, specificity, hallucinations, and more. We also show that fine-tuning with IIW data improves these metrics by +31% against models trained with prior work, even with only 9k samples. Lastly, we evaluate IIW models with text-to-image generation and vision-language reasoning tasks. Our generated descriptions result in the highest fidelity images, and boost compositional reasoning by up to 6% on ARO, SVO-Probes, and Winoground datasets. We release the IIW Eval benchmark with human judgement labels, object and image-level annotations from our framework, and existing image caption datasets enriched via IIW-model.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 5 May 2024.

Builds on

42 of 61 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2023PaLM 2 Technical Reportcited by 1,154
30 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  7. 2018nocaps: novel object captioning at scalecited by 444
  8. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  9. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  10. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  11. 2019Connecting Vision and Language with Localized Narrativescited by 225
  12. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  13. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  14. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  15. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  16. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  17. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  18. 2020Captioning Images Taken by People Who Are Blindcited by 108
  19. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  20. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  21. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  22. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  23. 2021Learning Compact Metrics for MTcited by 93
  24. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  25. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  26. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  27. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  28. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  29. 2019Quantifying the Impact of Cognitive Biases in Question-Answering Systemscited by 3
  30. 2023PixLore: A Dataset-driven Approach to Rich Image Captioningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 23; 2026: 16.

56 papers.

Sort:

By area:

  1. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  3. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 52
  4. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  5. 2024Baichuan-Omni Technical Reportcited by 42
  6. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 30
  7. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  8. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  9. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 23
  10. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  11. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  12. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
38 more
  1. 2025CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generationcited by 21
  2. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
  3. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  4. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  5. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  6. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  7. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
  8. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7
  9. 2025FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Textcited by 7
  10. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  11. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 6
  12. 2024Trust but Verify: Programmatic VLM Evaluation in the Wildcited by 5
  13. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captionscited by 5
  14. 2024Detailed Object Description with Controllable Dimensionscited by 3
  15. 2025Harnessing Caption Detailness for Data-Efficient Text-to-Image Generationcited by 3
  16. 2024BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphscited by 2
  17. 2024GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Designcited by 2
  18. 2025Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipelinecited by 2
  19. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  20. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMcited by 2
  21. 2026DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captionscited by 2
  22. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  23. 2025Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captionscited by 1
  24. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  25. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  26. 2026ReflectCAP: Detailed Image Captioning with Reflective Memorycited by 1
  27. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  28. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigationcited by 1
  29. 2024From Simple to Professional: A Combinatorial Controllable Image Captioning Agentcited by 0
  30. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Modelscited by 0
  31. 2025ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generationcited by 0
  32. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMscited by 0
  33. 2025Approximate Gaussian Mapping for Generative Image Steganographycited by 0
  34. 2025When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMscited by 0
  35. 2026Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Modelscited by 0
  36. 2026FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptionscited by 0
  37. 2026CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuningcited by 0
  38. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
  2. 2024A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models
  3. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  4. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  5. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  6. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  7. 2026Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
  8. 2024KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities