# The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

arXiv: https://arxiv.org/abs/2405.04623 · HTML: https://mlsanity.com/paper/2405.04623/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2024.

## Built on by (most cited first, 23 of 23)

- [LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions](https://mlsanity.com/paper/2406.08824/) (2024)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes](https://mlsanity.com/paper/2501.00962/) (2025)
- [Neglected Risks: The Disturbing Reality of Children's Images in Datasets and the Urgent Call for Accountability](https://mlsanity.com/paper/2504.14446/) (2025)
- [Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI](https://mlsanity.com/paper/2408.01959/) (2024)
- [Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models](https://mlsanity.com/paper/2510.03721/) (2025)
- [Sound Check: Auditing Audio Datasets](https://mlsanity.com/paper/2410.13114/) (2024)
- [Metamorphic Testing for Pose Estimation Systems](https://mlsanity.com/paper/2502.09460/) (2025)
- [SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge](https://mlsanity.com/paper/2511.16743/) (2025)
- [What Does 'Human-Centred AI' Mean?](https://mlsanity.com/paper/2507.19960/) (2025)
- [BiasMap: Leveraging Cross-Attentions to Discover and Mitigate Hidden Social Biases in Text-to-Image Generation](https://mlsanity.com/paper/2509.13496/) (2025)
- [RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation](https://mlsanity.com/paper/2502.07455/) (2025)
- [Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory](https://mlsanity.com/paper/2505.14080/) (2025)
- [Inference Time Debiasing Concepts in Diffusion Models](https://mlsanity.com/paper/2508.14933/) (2025)
- [Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning](https://mlsanity.com/paper/2511.01490/) (2025)
- [Do Large Language Models Reflect Demographic Pluralism in Safety?](https://mlsanity.com/paper/2602.07376/) (2026)
- [SEED-SET: Scalable Evolving Experimental Design for System-level Ethical Testing](https://mlsanity.com/paper/2603.01630/) (2026)
- [Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activities](https://mlsanity.com/paper/2504.06313/) (2025)
- [CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation](https://mlsanity.com/paper/2505.04851/) (2025)
- [Unmasking LAION-5B: Age, Gender, Race, and Emotion Biases in Large-Scale Image Datasets](https://mlsanity.com/paper/2606.23204/) (2026)
- [Autonomous VR-Based Risk Detection for Situational Awareness in Dangerous Settings](https://mlsanity.com/paper/2607.16582/) (2026)
- [Position: Fairness Failure in Generative Models is an Evaluation Problem](https://mlsanity.com/paper/2608.16974/) (2026)

## Abstract (arXiv)

Scale the model, scale the data, scale the GPU farms is the reigning sentiment in the world of generative AI today. While model scaling has been extensively studied, data scaling and its downstream impacts on model performance remain under-explored. This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the downstream impact of dataset scaling on 14 visio-linguistic models (VLMs) trained on the LAION400-M and LAION-2B datasets by measuring racial and gender bias using the Chicago Face Dataset (CFD) as the probe. Our results show that as the training data increased, the probability of a pre-trained CLIP model misclassifying human images as offensive non-human classes such as chimpanzee, gorilla, and orangutan decreased, but misclassifying the same images as human offensive classes such as criminal increased. Furthermore, of the 14 Vision Transformer-based VLMs we evaluated, the probability of predicting an image of a Black man and a Latino man as criminal increases by 65% and 69%, respectively, when the dataset is scaled from 400M to 2B samples for the larger ViT-L models. Conversely, for the smaller base ViT-B models, the probability of predicting an image of a Black man and a Latino man as criminal decreases by 20% and 47%, respectively, when the dataset is scaled from 400M to 2B samples. We ground the model audit results in a qualitative and historical analysis, reflect on our findings and their implications for dataset curation practice, and close with a summary of mitigation mechanisms and ways forward. Content warning: This article contains racially dehumanising and offensive descriptions.