mlsanityData to 14 Oct 2026

A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions

arXiv 2312.08578 ↗First posted 14 Dec 2023cs.CVcited by 108

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

31 of 47 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  11. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
19 more
  1. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  2. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  3. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  4. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  5. 2015YFCC100M: The New Data in Multimedia Researchcited by 235
  6. 2019Connecting Vision and Language with Localized Narrativescited by 226
  7. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  8. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  9. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  10. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  11. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  12. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  13. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  14. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  15. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  16. 2023CiT: Curation in Training for Effective Vision-Language Datacited by 27
  17. 2023Improved baselines for vision-language pre-trainingcited by 20
  18. 2018Enhancing Sentence Embedding with Generalized Poolingcited by 8
  19. 2023Mephisto: A Framework for Portable, Reproducible, and Iterative Crowdsourcingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 44; 2026: 32.

108 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2024The Platonic Representation Hypothesiscited by 382
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2024An Introduction to Vision-Language Modelingcited by 178
  5. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  6. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  7. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  8. 2024FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editingcited by 68
  9. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  10. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  11. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48
  12. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 38
38 more
  1. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 31
  2. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  3. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 28
  4. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  5. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  6. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 24
  7. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  8. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  9. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  10. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
  11. 2024TULIP: Token-length Upgraded CLIPcited by 19
  12. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 18
  13. 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Accelerationcited by 17
  14. 2025GOAL: Global-local Object Alignment Learningcited by 17
  15. 2025LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perceptioncited by 16
  16. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  17. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  18. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  19. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  20. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 13
  21. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 12
  22. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  23. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
  24. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 12
  25. 2024What Makes for Good Image Captions?cited by 11
  26. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  27. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 10
  28. 2025Adams Bashforth Moulton Solver for Inversion and Editing in Rectified Flowcited by 10
  29. 2024Multi-Modal Generative Embedding Modelcited by 7
  30. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7
  31. 2025FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Textcited by 7
  32. 2025Words That Make Language Models Perceivecited by 7
  33. 2025ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformerscited by 7
  34. 2024I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpaintingcited by 6
  35. 2024Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Taskscited by 6
  36. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  37. 2025The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Modelscited by 6
  38. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  2. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
  3. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  4. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  5. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  6. 2024Benchmarking and Improving Detail Image Caption
  7. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  8. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning