mlsanityData to 7 Oct 2026

Data Filtering Networks

arXiv 2309.17425 ↗First posted 29 Sep 2023cs.AI cs.LGcited by 247

Not on the map. Why?

Cited by 90 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large training sets have become a cornerstone of machine learning and are the foundation for recent advances in language modeling and multimodal learning. While data curation for pre-training is often still ad-hoc, one common paradigm is to first collect a massive pool of data from the Web and then filter this candidate pool down to an actual training set via various heuristics. In this work, we study the problem of learning a data filtering network (DFN) for this second step of filtering a large uncurated dataset. Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks: for instance, a model that performs well on ImageNet can yield worse training sets than a model with low ImageNet accuracy that is trained on a small amount of high-quality data. Based on our insights, we construct new data filtering networks that induce state-of-the-art image-text datasets. Specifically, our best performing dataset DFN-5B enables us to train state-of-the-art CLIP models for their compute budgets: among other improvements on a variety of tasks, a ViT-H trained on our dataset achieves 84.4% zero-shot transfer accuracy on ImageNet, out-performing models trained on other datasets such as LAION-2B, DataComp-1B, or OpenAI's WIT. In order to facilitate further research in dataset design, we also release a new 2 billion example dataset DFN-2B and show that high performance data filtering networks can be trained from scratch using only publicly available data.

Key facts. Cited by 247 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

32 of 47 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
20 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2021Scaling Vision Transformerscited by 993
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2023Demystifying CLIP Datacited by 265
  12. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  13. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  14. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  15. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  16. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  17. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  18. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  19. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  20. 2023CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a 10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracycited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 81; 2025: 106; 2026: 54.

247 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  5. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  7. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  9. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  10. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  11. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  12. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
38 more
  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  2. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  3. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  4. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  5. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  6. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  7. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  8. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
  9. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  10. 2025Scaling Language-Free Visual Representation Learningcited by 62
  11. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  12. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  13. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  14. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  15. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  16. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  17. 2025Scaling Laws for Optimal Data Mixturescited by 49
  18. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  19. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 47
  20. 2025Scaling Laws for Native Multimodal Modelscited by 47
  21. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 43
  22. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  23. 2024Baichuan-Omni Technical Reportcited by 42
  24. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  25. 2025CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AIcited by 42
  26. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  27. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  28. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  29. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  30. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  31. 2023Sieve: Multimodal Dataset Pruning Using Image Captioning Modelscited by 33
  32. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  33. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  34. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 33
  35. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  36. 2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
  37. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  38. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29

50 most cited of 247 All 247 (JSON)