mlsanityData to 7 Oct 2026

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

arXiv 2303.09540 ↗First posted 16 Mar 2023cs.LG cs.AI cs.CVcited by 295

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.

Key facts. Cited by 295 papers in the map (data through 7 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

33 of 49 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
21 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2021Scaling Vision Transformerscited by 993
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  7. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  8. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  10. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  11. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  12. 2021Scaling Laws for Transfercited by 265
  13. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  14. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  15. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  16. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  17. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  18. 2016Coresets and Sketchescited by 63
  19. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  20. 2022Noise-Robust De-Duplication at Scalecited by 17
  21. 2022DUEL: Adaptive Duplicate Elimination on Working Memory for Self-Supervised Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 35; 2024: 95; 2025: 89; 2026: 76.

295 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  5. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  7. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  8. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  9. 2023Demystifying CLIP Datacited by 265
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 169
38 more
  1. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  2. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  3. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  4. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  5. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  6. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  7. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  8. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  9. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  10. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  11. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  12. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  13. 2024How to Train Data-Efficient LLMscited by 91
  14. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  15. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  16. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  17. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  18. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  19. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  20. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  21. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  22. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  23. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
  24. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  25. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  26. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  27. 2023A Reproducible Extraction of Training Images from Diffusion Modelscited by 60
  28. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  29. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  30. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  31. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  32. 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
  33. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  34. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  35. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  36. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  37. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  38. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 38

50 most cited of 295 All 295 (JSON)