mlsanityData to 7 Oct 2026

What If We Recaption Billions of Web Images with LLaMA-3?

arXiv 2406.08478 ↗First posted 12 Jun 2024cs.CV cs.CLcited by 82

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate that semantically aligning and enriching textual descriptions of these pairs can significantly enhance model training across various vision-language tasks, particularly text-to-image generation. However, large-scale investigations in this area remain predominantly closed-source. Our paper aims to bridge this community effort, leveraging the powerful and textitopen-sourced LLaMA-3, a GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images from the DataComp-1B dataset. Our empirical results confirm that this enhanced dataset, Recap-DataComp-1B, offers substantial benefits in training advanced vision-language models. For discriminative models like CLIP, we observe enhanced zero-shot performance in cross-modal retrieval tasks. For generative models like text-to-image Diffusion Transformers, the generated images exhibit a significant improvement in alignment with users' text instructions, especially in following complex queries. Our project page is https://www.haqtu.me/Recap-Datacomp-1B/

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 58 of its 73 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

58 of 73 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  15. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  16. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  17. 2023Adversarial Diffusion Distillationcited by 741
  18. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  19. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  21. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  22. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  23. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  24. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  25. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  26. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  27. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  28. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  29. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  30. 2023Demystifying CLIP Datacited by 265
  31. 2023Data Filtering Networkscited by 247
  32. 2023Improving CLIP Training with Language Rewritescited by 216
  33. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  34. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  35. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 191
  36. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  37. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  38. 2023Improving Multimodal Datasets with Image Captioningcited by 93

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 48; 2026: 13.

82 papers.

Sort:

By area:

  1. 2024OmniGen: Unified Image Generationcited by 388
  2. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  3. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  4. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  5. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  6. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  7. 2025Unified Multimodal Discrete Diffusioncited by 46
  8. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  9. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  10. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Modelscited by 37
  11. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  12. 2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
38 more
  1. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  2. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  3. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  4. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  5. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  6. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  7. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 20
  8. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20
  9. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  10. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  11. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  12. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  13. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  14. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 15
  15. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Modelscited by 14
  16. 2025SmartCLIP: Modular Vision-language Alignment with Identification Guaranteescited by 14
  17. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  18. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
  19. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  20. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11
  21. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustnesscited by 10
  22. 2025How far can we go with ImageNet for Text-to-Image generation?cited by 10
  23. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  24. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  25. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Modelscited by 9
  26. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captionscited by 8
  27. 2024AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Modelscited by 7
  28. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  29. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7
  30. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learningcited by 7
  31. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  32. 2025Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scalecited by 6
  33. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  34. 2025Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-trainingcited by 5
  35. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  36. 2026OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generationcited by 5
  37. 2025Vision-Speech Models: Teaching Speech Models to Converse about Imagescited by 4
  38. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  2. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions
  3. 2023Linear Alignment of Vision-language Models for Image Captioning
  4. 2025Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
  6. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  7. 2024Boosting Alignment for Post-Unlearning Text-to-Image Generative Models
  8. 2022CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge