# A Decade's Battle on Dataset Bias: Are We There Yet?

arXiv: https://arxiv.org/abs/2403.08632 · HTML: https://mlsanity.com/paper/2403.08632/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 74 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)

## Built on by (most cited first, 30 of 47)

- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [A Bias-Free Training Paradigm for More General AI-generated Image Detection](https://mlsanity.com/paper/2412.17671/) (2024)
- [Scaling Language-Free Visual Representation Learning](https://mlsanity.com/paper/2504.01017/) (2025)
- [LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Model](https://mlsanity.com/paper/2405.02363/) (2024)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation](https://mlsanity.com/paper/2506.16802/) (2025)
- [Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights](https://mlsanity.com/paper/2506.16406/) (2025)
- [Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models](https://mlsanity.com/paper/2501.07972/) (2025)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [Understanding Bias in Large-Scale Visual Datasets](https://mlsanity.com/paper/2412.01876/) (2024)
- [ImageNot: A contrast with ImageNet preserves model rankings](https://mlsanity.com/paper/2404.02112/) (2024)
- [Always Skip Attention](https://mlsanity.com/paper/2505.01996/) (2025)
- [In Pursuit of Pixel Supervision for Visual Pre-training](https://mlsanity.com/paper/2512.15715/) (2025)
- [Accessing Vision Foundation Models via ImageNet-1K](https://mlsanity.com/paper/2407.10366/) (2024)
- [BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models](https://mlsanity.com/paper/2408.04556/) (2024)
- [Natural Fingerprints of Large Language Models](https://mlsanity.com/paper/2504.14871/) (2025)
- [Shortcut Learning in Binary Classifier Black Boxes: Applications to Voice Anti-Spoofing and Biometrics](https://mlsanity.com/paper/2601.17782/) (2026)
- [Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers?](https://mlsanity.com/paper/2405.18029/) (2024)
- [Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Training](https://mlsanity.com/paper/2412.02857/) (2024)
- [DATA: Multi-Disentanglement based Contrastive Learning for Open-World Semi-Supervised Deepfake Attribution](https://mlsanity.com/paper/2505.04384/) (2025)
- [ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts](https://mlsanity.com/paper/2510.26186/) (2025)
- [Modeling Saliency Dataset Bias](https://mlsanity.com/paper/2505.10169/) (2025)
- [Processing and acquisition traces in visual encoders: What does CLIP know about your camera?](https://mlsanity.com/paper/2508.10637/) (2025)
- [NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired People](https://mlsanity.com/paper/2410.18109/) (2024)
- [BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance](https://mlsanity.com/paper/2506.03589/) (2025)
- [Limitations of Public Chest Radiography Datasets for Artificial Intelligence: Label Quality, Domain Shift, Bias and Evaluation Challenges](https://mlsanity.com/paper/2509.15107/) (2025)
- [Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition](https://mlsanity.com/paper/2510.26466/) (2025)
- [Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models](https://mlsanity.com/paper/2410.03640/) (2024)

## Abstract (arXiv)

We revisit the "dataset classification" experiment suggested by Torralba & Efros (2011) a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we observe that modern neural networks can achieve excellent accuracy in classifying which dataset an image is from: e.g., we report 84.7% accuracy on held-out validation data for the three-way classification problem consisting of the YFCC, CC, and DataComp datasets. Our further experiments show that such a dataset classifier could learn semantic features that are generalizable and transferable, which cannot be explained by memorization. We hope our discovery will inspire the community to rethink issues involving dataset bias.