mlsanityData to 7 Oct 2026

Improving Multimodal Datasets with Image Captioning

arXiv 2307.10350 ↗First posted 19 Jul 2023cs.LG cs.CVcited by 93

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a candidate pool of 128M image-text pairs. Our best approach is also 2x better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. In experimenting with different image captioning models, we also demonstrate that the performance of a model on standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable indicator of the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used in our experiments are now available on HuggingFace.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 58 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

47 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
31 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2017VisDA: The Visual Domain Adaptation Challengecited by 432
  11. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  12. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  13. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  14. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  15. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  16. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  17. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  18. 2023Improving CLIP Training with Language Rewritescited by 216
  19. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  20. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  21. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  22. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  23. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
  24. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  25. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  26. 2019FairFace: Face Attribute Dataset for Balanced Race, Gender, and Agecited by 87
  27. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  28. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  29. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  30. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  31. 2023Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustnesscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 33; 2025: 31; 2026: 10.

93 papers.

Sort:

By area:

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  4. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  5. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  6. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  7. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  8. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  9. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  10. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  11. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 72
  12. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
38 more
  1. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  2. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  3. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 40
  4. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  5. 2023Sieve: Multimodal Dataset Pruning Using Image Captioning Modelscited by 33
  6. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  7. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  8. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  9. 2024Distilling Vision-Language Models on Millions of Videoscited by 20
  10. 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20
  11. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20
  12. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19
  13. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  14. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 18
  15. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  16. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  17. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  18. 2023CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Imagescited by 15
  19. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  20. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  21. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  22. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 11
  23. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  24. 2023MAFA: Managing False Negatives for Vision-Language Pre-trainingcited by 9
  25. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  26. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Modelscited by 9
  27. 2023GIST: Generating Image-Specific Text for Fine-grained Object Classificationcited by 8
  28. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  29. 2023Towards Open-Ended Visual Recognition with Large Language Modelcited by 8
  30. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captionscited by 8
  31. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  32. 2023Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute associationcited by 7
  33. 2023CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languagescited by 7
  34. 2024Curriculum Learning with Quality-Driven Data Selectioncited by 7
  35. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7
  36. 2024HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understandingcited by 7
  37. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  38. 2026CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioningcited by 7

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
  2. 2023Image Captioning with Multi-Context Synthetic Data
  3. 2024CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions
  4. 2026VIVECaption: A Split Approach to Caption Quality Improvement
  5. 2024Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model
  6. 2023DiffCap: Exploring Continuous Diffusion on Image Captioning
  7. 2023Multimodal Data Augmentation for Image Captioning using Diffusion Models
  8. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning