mlsanityData to 14 Oct 2026

Data Leakage in Visual Datasets

arXiv 2508.17416 ↗First posted 24 Aug 2025cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

We analyze data leakage in visual datasets. Data leakage refers to images in evaluation benchmarks that have been seen during training, compromising fair model evaluation. Given that large-scale datasets are often sourced from the internet, where many computer vision benchmarks are publicly available, our efforts are focused into identifying and studying this phenomenon. We characterize visual leakage into different types according to its modality, coverage, and degree. By applying image retrieval techniques, we unequivocally show that all the analyzed datasets present some form of leakage, and that all types of leakage, from severe instances to more subtle cases, compromise the reliability of model evaluation in downstream tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

44 of 60 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2015Fast R-CNNcited by 2,933
32 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  10. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  11. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  12. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  13. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  14. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  15. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 425
  16. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  17. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  18. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  19. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  20. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  21. 2022Data Contamination: From Memorization to Exploitationcited by 236
  22. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  23. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  24. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  25. 2015Particular object retrieval with integral max-pooling of CNN activationscited by 161
  26. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  27. 2023What's In My Big Data?cited by 129
  28. 2023Into the LAIONs Den: Investigating Hate in Multimodal Datasetscited by 54
  29. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 52
  30. 2022Gender Artifacts in Visual Datasetscited by 38
  31. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 31
  32. 2024Would Deep Generative Models Amplify Bias in Future Models?cited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  2. 2025PubTables-v2: A new large-scale dataset for full-page and multi-page table extractioncited by 2
  3. 2026Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representationscited by 1
  4. 2026Deep Learning-Based Meat Freshness Detection with Segmentation and OOD-Aware Classificationcited by 0
  5. 2026FecalFed: Privacy-Preserving Poultry Disease Detection via Federated Learningcited by 0
  6. 2026Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Modelscited by 0
  7. 2026Cost-Aware Best-LLM Identification using Dueling Feedbackcited by 0