# SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification arXiv: https://arxiv.org/abs/2410.05057 ยท HTML: https://mlsanity.com/paper/2410.05057/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 50 references that are in the map. - First posted on arXiv 7 Oct 2024. ## Builds on (most cited first, 30 of 31) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021) - [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018) - [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019) - [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019) - [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022) - [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023) - [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020) - [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021) - [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021) - [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023) - [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021) - [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022) - [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022) - [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019) - [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022) - [Leaving Reality to Imagination: Robust Classification via Generated Datasets](https://mlsanity.com/paper/2302.02503/) (2023) - [Gender and Racial Bias in Visual Question Answering Datasets](https://mlsanity.com/paper/2205.08148/) (2022) - [On the Connection between Pre-training Data Diversity and Fine-tuning Robustness](https://mlsanity.com/paper/2307.12532/) (2023) - [Bugs in the Data: How ImageNet Misrepresents Biodiversity](https://mlsanity.com/paper/2208.11695/) (2022) - [ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object](https://mlsanity.com/paper/2403.18775/) (2024) - [Simplifying Neural Network Training Under Class Imbalance](https://mlsanity.com/paper/2312.02517/) (2023) ## Built on by (most cited first, 5 of 5) - [DataS^3: Dataset Subset Selection for Specialization](https://mlsanity.com/paper/2504.16277/) (2025) - [WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training](https://mlsanity.com/paper/2501.18511/) (2025) - [Attribution-Guided Model Rectification of Unreliable Neural Network Behaviors](https://mlsanity.com/paper/2603.15656/) (2026) - [Beyond the Golden Data: Resolving the Motion-Vision Quality Dilemma via Timestep Selective Training](https://mlsanity.com/paper/2603.25527/) (2026) - [Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores](https://mlsanity.com/paper/2512.01080/) (2025) ## Abstract (arXiv) Data curation is the problem of how to collect and organize samples into a dataset that supports efficient learning. Despite the centrality of the task, little work has been devoted towards a large-scale, systematic comparison of various curation methods. In this work, we take steps towards a formal evaluation of data curation strategies and introduce SELECT, the first large-scale benchmark of curation strategies for image classification. In order to generate baseline methods for the SELECT benchmark, we create a new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K to date. Our dataset extends ImageNet with 5 new training-data shifts, each approximately the size of ImageNet-1K itself, and each assembled using a distinct curation strategy. We evaluate our data curation baselines in two ways: (i) using each training-data shift to train identical image classification models from scratch (ii) using the data itself to fit a pretrained self-supervised representation. Our findings show interesting trends, particularly pertaining to recent methods for data curation such as synthetic data generation and lookup based on CLIP embeddings. We show that although these strategies are highly competitive for certain tasks, the curation strategy used to assemble the original ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark can illuminate the path for new methods to further reduce the gap. We release our checkpoints, code, documentation, and a link to our dataset at https://github.com/jimmyxu123/SELECT.