mlsanityData to 14 Oct 2026

DOCCI: Descriptions of Connected and Contrasting Images

arXiv 2404.19753 ↗First posted 30 Apr 2024cs.CV cs.AI cs.CLcited by 132

Not on the map. Why?

Cited by 61 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-language datasets are vital for both text-to-image (T2I) and image-to-text (I2T) research. However, current datasets lack descriptions with fine-grained detail that would allow for richer associations to be learned by models. To fill the gap, we introduce Descriptions of Connected and Contrasting Images (DOCCI), a dataset with long, human-annotated English descriptions for 15k images that were taken, curated and donated by a single researcher intent on capturing key challenges such as spatial relations, counting, text rendering, world knowledge, and more. We instruct human annotators to create comprehensive descriptions for each image; these average 136 words in length and are crafted to clearly distinguish each image from those that are related or similar. Each description is highly compositional and typically encompasses multiple challenges. Through both quantitative and qualitative analyses, we demonstrate that DOCCI serves as an effective training resource for image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or superior results compared to highly-performant larger models like LLaVA-1.5 7B and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for text-to-image generation, highlighting the limitations of current text-to-image models in capturing long descriptions and fine details.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). Builds on 44 of its 65 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

44 of 65 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
32 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2023PaLM 2 Technical Reportcited by 1,162
  8. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  11. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  12. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  13. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  14. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  15. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  16. 2019Connecting Vision and Language with Localized Narrativescited by 226
  17. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  18. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  19. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
  20. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  21. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  22. 2020BLEU might be Guilty but References are not Innocentcited by 110
  23. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  24. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  25. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  26. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  27. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 95
  28. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  29. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 68
  30. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  31. 2021Transparent Human Evaluation for Image Captioningcited by 40
  32. 2022Underspecification in Scene Description-to-Depiction Taskscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 38; 2025: 55; 2026: 39.

132 papers.

Sort:

By area:

  1. 2024OmniGen: Unified Image Generationcited by 394
  2. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  3. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  4. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  6. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 73
  7. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
  8. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  9. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48
  10. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 41
  11. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  12. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
38 more
  1. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 31
  2. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 31
  3. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 27
  4. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  5. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  6. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 22
  7. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  8. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
  9. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 20
  10. 2024TULIP: Token-length Upgraded CLIPcited by 19
  11. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  12. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  13. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  14. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  15. 2025GOAL: Global-local Object Alignment Learningcited by 17
  16. 2025LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perceptioncited by 16
  17. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  18. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  19. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  20. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  21. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  22. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 14
  23. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 13
  24. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  25. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  26. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Modelscited by 12
  27. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
  28. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  29. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  30. 2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
  31. 2024MATE: Meet At The Embedding -- Connecting Images with Long Textscited by 10
  32. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  33. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 10
  34. 2025OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detectioncited by 10
  35. 2026Gemini Embedding 2: A Native Multimodal Embedding Model from Geminicited by 10
  36. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  37. 2026WeMM-Embedding: WeChat Multi-Modal Embedding Technical Reportcited by 8
  38. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7

50 most cited of 132 All 132 (JSON)