On the Difficulty of Learning a Meta-network for Training Data Selection
Not on the map. Why?
Abstract (arXiv)
Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four benchmarks show consistent improvements, achieving average gains of 5.49% over training without selection and 2.89% over the strongest baseline.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 58 of its 69 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
58 of 69 references.
References · 58
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
- 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
- 2017Deeper, Broader and Artier Domain Generalizationcited by 986
- 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
- 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
- 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
- 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
- 2023Data Selection for Language Models via Importance Resamplingcited by 329
- 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
- 2022Is synthetic data from generative models ready for image recognition?cited by 320
38 more
- 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
- 2023Effective Data Augmentation With Diffusion Modelscited by 283
- 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
- 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
- 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
- 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
- 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 108
- 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
- 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
- 2018Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clusteringcited by 94
- 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
- 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
- 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
- 2024DiffuseMix: Label-Preserving Data Augmentation with Diffusion Modelscited by 78
- 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
- 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
- 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
- 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
- 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
- 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
- 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
- 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
- 2023MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentationcited by 35
- 2024Enhance Image Classification via Inter-Class Image Mixup with Diffusion Modelcited by 34
- 2021Featurized Density Ratio Estimationcited by 33
- 2024LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Contentcited by 33
- 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
- 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
- 2024DataDream: Few-shot Guided Dataset Generationcited by 29
- 2024TSDS: Data Selection for Task-Specific Model Finetuningcited by 28
- 2024InstaGen: Enhancing Object Detection by Training on Synthetic Datasetcited by 26
- 2021The effective noise of Stochastic Gradient Descentcited by 25
- 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
- 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20
- 2022Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generatorscited by 17
- 2023Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classificationcited by 16
- 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
- 2020Meta-Semi: A Meta-learning Approach for Semi-supervised Learningcited by 11
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.