# A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions

arXiv: https://arxiv.org/abs/2312.08578 · HTML: https://mlsanity.com/paper/2312.08578/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](https://mlsanity.com/paper/2204.08790/) (2022)
- [VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations](https://mlsanity.com/paper/2207.00221/) (2022)
- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [An Inverse Scaling Law for CLIP Training](https://mlsanity.com/paper/2305.07017/) (2023)
- [Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models](https://mlsanity.com/paper/2305.19595/) (2023)
- [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning](https://mlsanity.com/paper/2308.03977/) (2023)
- [CiT: Curation in Training for Effective Vision-Language Data](https://mlsanity.com/paper/2301.02241/) (2023)
- [Improved baselines for vision-language pre-training](https://mlsanity.com/paper/2305.08675/) (2023)
- [Enhancing Sentence Embedding with Generalized Pooling](https://mlsanity.com/paper/1806.09828/) (2018)

## Built on by (most cited first, 30 of 108)

- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://mlsanity.com/paper/2411.05007/) (2024)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025)
- [FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing](https://mlsanity.com/paper/2412.07517/) (2024)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [From Pixels to Prose: A Large Dataset of Dense Image Captions](https://mlsanity.com/paper/2406.10328/) (2024)
- [FLAIR: VLM with Fine-grained Language-informed Image Representations](https://mlsanity.com/paper/2412.03561/) (2024)
- [PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training](https://mlsanity.com/paper/2503.06486/) (2025)
- [Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning](https://mlsanity.com/paper/2412.08614/) (2024)
- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)
- [Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP](https://mlsanity.com/paper/2501.10913/) (2025)
- [LoTLIP: Improving Language-Image Pre-training for Long Text Understanding](https://mlsanity.com/paper/2410.05249/) (2024)
- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Fantastic Copyrighted Beasts and How (Not) to Generate Them](https://mlsanity.com/paper/2406.14526/) (2024)
- [FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs](https://mlsanity.com/paper/2504.01916/) (2025)
- [Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences](https://mlsanity.com/paper/2506.02095/) (2025)
- [MobileCLIP2: Improving Multi-Modal Reinforced Training](https://mlsanity.com/paper/2508.20691/) (2025)
- [Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage](https://mlsanity.com/paper/2412.15484/) (2024)
- [TULIP: Token-length Upgraded CLIP](https://mlsanity.com/paper/2410.10034/) (2024)
- [A Survey on Quality Metrics for Text-to-Image Generation](https://mlsanity.com/paper/2403.11821/) (2024)
- [HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration](https://mlsanity.com/paper/2410.01723/) (2024)
- [GOAL: Global-local Object Alignment Learning](https://mlsanity.com/paper/2503.17782/) (2025)
- [LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception](https://mlsanity.com/paper/2504.15362/) (2025)
- [SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning](https://mlsanity.com/paper/2508.06125/) (2025)
- [COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training](https://mlsanity.com/paper/2412.01814/) (2024)
- [ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing](https://mlsanity.com/paper/2506.19848/) (2025)

## Abstract (arXiv)

Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come.