# A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances

arXiv: https://arxiv.org/abs/2505.17799 · HTML: https://mlsanity.com/paper/2505.17799/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 118 of its 178 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 118)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)

## Built on by (most cited first, 28 of 28)

- [SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone](https://mlsanity.com/paper/2510.17998/) (2025)
- [A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)](https://mlsanity.com/paper/2602.14696/) (2026)
- [VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction](https://mlsanity.com/paper/2511.18831/) (2025)
- [DUSE: A Data Expansion Framework for Low-resource Automatic Modulation Recognition based on Active Learning](https://mlsanity.com/paper/2507.12011/) (2025)
- [Bound to Disagree: Generalization Bounds via Certifiable Surrogates](https://mlsanity.com/paper/2602.23128/) (2026)
- [Efficient Benchmarking Is Just Feature Selection and Multiple Regression](https://mlsanity.com/paper/2605.25773/) (2026)
- [Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution](https://mlsanity.com/paper/2505.21438/) (2025)
- [ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios](https://mlsanity.com/paper/2602.16073/) (2026)
- [Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning](https://mlsanity.com/paper/2606.25488/) (2026)
- [Iterative Misclassification Error Training (IMET): An Optimized Neural Network Training Technique for Image Classification](https://mlsanity.com/paper/2507.02979/) (2025)
- [SubZeroCore: A Submodular Approach with Zero Training for Coreset Selection](https://mlsanity.com/paper/2509.21748/) (2025)
- [Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Method](https://mlsanity.com/paper/2510.21286/) (2025)
- [Learning from Complexity: Exploring Dynamic Sample Pruning of Spatio-Temporal Training](https://mlsanity.com/paper/2602.19113/) (2026)
- [Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AI](https://mlsanity.com/paper/2602.19789/) (2026)
- [A Dataset is Worth 1 MB](https://mlsanity.com/paper/2602.23358/) (2026)
- [SAS: Semantic-aware Sampling for Generative Dataset Distillation](https://mlsanity.com/paper/2605.18012/) (2026)
- [Stein Kernelized Molecular Dynamics for Active Learning of Interatomic Potentials](https://mlsanity.com/paper/2606.04100/) (2026)
- [Group-invariant Coresets for Data-efficient Active Learning](https://mlsanity.com/paper/2607.01089/) (2026)
- [Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation](https://mlsanity.com/paper/2607.05891/) (2026)
- [A Coreset Selection Framework with Ensemble Aggregation for Image Classification](https://mlsanity.com/paper/2607.09100/) (2026)
- [Selectivity Drives Efficiency: Dataset Pruning for Visual Place Recognition](https://mlsanity.com/paper/2607.14897/) (2026)
- [ERank in Latent Space as an Image-Complexity and Richness Measure](https://mlsanity.com/paper/2607.19315/) (2026)
- [Dataset Distillation Based on Saliency-Driven Prototype Alignment](https://mlsanity.com/paper/2607.25318/) (2026)
- [First-order Constrained Trilevel Optimization Over Distributed Networks for Robust Coreset Selection](https://mlsanity.com/paper/2607.27632/) (2026)
- [EMS Coreset: An Efficient Expectation-Maximization Algorithm for Sinkhorn Coreset](https://mlsanity.com/paper/2608.16101/) (2026)
- [Fidelity-Diversity-Consistency (FDC): Data Pruning for Remote Sensing Change Detection](https://mlsanity.com/paper/2608.21754/) (2026)
- [A budget-dependent crossover between coverage- and response-based training-set selection for machine-learned interatomic potentials](https://mlsanity.com/paper/2609.05877/) (2026)
- [Dataset Pruning from First Principles: A Label-Free Linear Programming Approach](https://mlsanity.com/paper/2610.10347/) (2026)

## Abstract (arXiv)

Coreset selection targets the challenge of finding a small, representative subset of a large dataset that preserves essential patterns for effective machine learning. Although several surveys have examined data reduction strategies before, most focus narrowly on either classical geometry-based methods or active learning techniques. In contrast, this survey presents a more comprehensive view by unifying three major lines of coreset research, namely, training-free, training-oriented, and label-free approaches, into a single taxonomy. We present subfields often overlooked by existing work, including submodular formulations, bilevel optimization, and recent progress in pseudo-labeling for unlabeled datasets. Additionally, we examine how pruning strategies influence generalization and neural scaling laws, offering new insights that are absent from prior reviews. Finally, we compare these methods under varying computational, robustness, and performance demands and highlight open challenges, such as robustness, outlier filtering, and adapting coreset selection to foundation models, for future research.