# Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond

arXiv: https://arxiv.org/abs/2408.11338 · HTML: https://mlsanity.com/paper/2408.11338/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 78 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning](https://mlsanity.com/paper/1803.04765/) (2018)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Learning with Instance-Dependent Label Noise: A Sample Sieve Approach](https://mlsanity.com/paper/2010.02347/) (2020)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)

## Built on by (most cited first, 18 of 18)

- [LLM Unlearning via Loss Adjustment with Only Forget Data](https://mlsanity.com/paper/2410.11143/) (2024)
- [Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning](https://mlsanity.com/paper/2505.11953/) (2025)
- [OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning](https://mlsanity.com/paper/2410.18963/) (2024)
- [Improving Data Efficiency via Curating LLM-Driven Rating Systems](https://mlsanity.com/paper/2410.10877/) (2024)
- [Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models](https://mlsanity.com/paper/2506.14674/) (2025)
- [AutoData: A Multi-Agent System for Open Web Data Collection](https://mlsanity.com/paper/2505.15859/) (2025)
- [GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection](https://mlsanity.com/paper/2505.13312/) (2025)
- [Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring](https://mlsanity.com/paper/2506.08429/) (2025)
- [When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification](https://mlsanity.com/paper/2505.16149/) (2025)
- [SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixing](https://mlsanity.com/paper/2509.11265/) (2025)
- [GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning](https://mlsanity.com/paper/2505.22661/) (2025)
- [LM-mixup: Text Data Augmentation via Language Model based Mixup](https://mlsanity.com/paper/2510.20449/) (2025)
- [Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth](https://mlsanity.com/paper/2506.06991/) (2025)
- [Small-Margin Preferences Still Matter-If You Train Them Right](https://mlsanity.com/paper/2602.00954/) (2026)
- [What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media](https://mlsanity.com/paper/2606.06784/) (2026)
- [GenIR: Generative Visual Feedback for Mental Image Retrieval](https://mlsanity.com/paper/2506.06220/) (2025)
- [Incentivizing High-quality Participation From Federated Learning Agents](https://mlsanity.com/paper/2506.16731/) (2025)
- [Supplement Generation Training for Enhancing Agentic Task Performance](https://mlsanity.com/paper/2604.20727/) (2026)

## Abstract (arXiv)

Large-scale data collection is essential for developing personalized training data, mitigating the shortage of training data, and fine-tuning specialized models. However, creating high-quality datasets quickly and accurately remains a challenge due to annotation errors, the substantial time and costs associated with human labor. To address these issues, we propose Automatic Dataset Construction (ADC), an innovative methodology that automates dataset creation with negligible cost and high efficiency. Taking the image classification task as a starting point, ADC leverages LLMs for the detailed class design and code generation to collect relevant samples via search engines, significantly reducing the need for manual annotation and speeding up the data generation process. To demonstrate ADC at scale, we construct Clothing-ADC: a dataset of over 1 million images spanning 12 main classes and 12,000 fine-grained subclasses. Our automated curation achieves 79% agreement with human annotators and reduces label noise from 22.2% to 10.7%. Despite these advantages, ADC also encounters real-world challenges such as label errors (label noise) and imbalanced data distributions (label bias). We provide open-source software that incorporates existing methods for label error detection, robust learning under noisy and biased data, ensuring a higher-quality training data and more robust model training procedure. Furthermore, we design three benchmark datasets focused on label noise detection, label noise learning, and class-imbalanced learning. These datasets are vital because there are few existing datasets specifically for label noise detection, despite its importance. Finally, we evaluate the performance of existing popular methods on these datasets, thereby facilitating further research in the field.