# On the De-duplication of LAION-2B arXiv: https://arxiv.org/abs/2303.12733 ยท HTML: https://mlsanity.com/paper/2303.12733/ ## Key facts - Cited by 47 papers in the map (data through 14 Oct 2026). - Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. - Builds on 16 of its 25 references that are in the map. - First posted on arXiv 17 Mar 2023. ## Placed in - Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/) ## Builds on (most cited first, 16 of 16) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016) - [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023) - [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022) - [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017) - [StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis](https://mlsanity.com/paper/2301.09515/) (2023) - [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022) - [A Self-Supervised Descriptor for Image Copy Detection](https://mlsanity.com/paper/2202.10261/) (2022) - [A Survey on Locality Sensitive Hashing Algorithms and their Applications](https://mlsanity.com/paper/2102.08942/) (2021) - [CE-Dedup: Cost-Effective Convolutional Neural Nets Training based on Image Deduplication](https://mlsanity.com/paper/2109.00899/) (2021) ## Built on by (most cited first, 30 of 47) - [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023) - [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023) - [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023) - [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024) - [Unveiling Encoder-Free Vision-Language Models](https://mlsanity.com/paper/2406.11832/) (2024) - [WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens](https://mlsanity.com/paper/2401.09985/) (2024) - [An Inverse Scaling Law for CLIP Training](https://mlsanity.com/paper/2305.07017/) (2023) - [DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception](https://mlsanity.com/paper/2407.08303/) (2024) - [Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks](https://mlsanity.com/paper/2310.19909/) (2023) - [A Reproducible Extraction of Training Images from Diffusion Models](https://mlsanity.com/paper/2305.08694/) (2023) - [Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention](https://mlsanity.com/paper/2403.11052/) (2024) - [ControlVAR: Exploring Controllable Visual Autoregressive Modeling](https://mlsanity.com/paper/2406.09750/) (2024) - [Copyright Protection in Generative AI: A Technical Perspective](https://mlsanity.com/paper/2402.02333/) (2024) - [The Evolution of Multimodal Model Architectures](https://mlsanity.com/paper/2405.17927/) (2024) - [The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering](https://mlsanity.com/paper/2309.15954/) (2023) - [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024) - [DASH: Detection and Assessment of Systematic Hallucinations of VLMs](https://mlsanity.com/paper/2503.23573/) (2025) - [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024) - [Generalization of Diffusion Models Arises with a Balanced Representation Space](https://mlsanity.com/paper/2512.20963/) (2025) - [Slight Corruption in Pre-training Data Makes Better Diffusion Models](https://mlsanity.com/paper/2405.20494/) (2024) - [mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus](https://mlsanity.com/paper/2406.08707/) (2024) - [U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AI](https://mlsanity.com/paper/2406.15386/) (2024) - [An Inversion-based Measure of Memorization for Diffusion Models](https://mlsanity.com/paper/2405.05846/) (2024) - [SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency](https://mlsanity.com/paper/2411.09126/) (2024) - [Neglected Risks: The Disturbing Reality of Children's Images in Datasets and the Urgent Call for Accountability](https://mlsanity.com/paper/2504.14446/) (2025) - [How Diffusion Models Memorize](https://mlsanity.com/paper/2509.25705/) (2025) - [Memory Triggers: Unveiling Memorization in Text-To-Image Generative Models through Word-Level Duplication](https://mlsanity.com/paper/2312.03692/) (2023) - [MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models](https://mlsanity.com/paper/2407.17095/) (2024) - [AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation](https://mlsanity.com/paper/2411.04967/) (2024) - [MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection](https://mlsanity.com/paper/2405.19458/) (2024) ## Abstract (arXiv) Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million images, or about 30%, of LAION-2B's images are likely duplicated. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set will be distributed online.