mlsanityData to 7 Oct 2026

Data Summarization via Bilevel Optimization

arXiv 2109.12534 ↗First posted 26 Sep 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

The increasing availability of massive data sets poses a series of challenges for machine learning. Prominent among these is the need to learn models under hardware or human resource constraints. In such resource-constrained settings, a simple yet powerful approach is to operate on small subsets of the data. Coresets are weighted subsets of the data that provide approximation guarantees for the optimization objective. However, existing coreset constructions are highly model-specific and are limited to simple models such as linear regression, logistic regression, and k-means. In this work, we propose a generic coreset construction framework that formulates the coreset selection as a cardinality-constrained bilevel optimization problem. In contrast to existing approaches, our framework does not require model-specific adaptations and applies to any twice differentiable model, including neural networks. We show the effectiveness of our framework for a wide range of models in various settings, including training non-convex models online and batch active learning.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 48 of its 98 references that are in the map. First posted on arXiv 26 Sep 2021.

Builds on

48 of 98 references.

References · 48

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  12. 2017Continual Learning with Deep Generative Replaycited by 1,152
36 more
  1. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  3. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2019Gradient based sample selection for online continual learningcited by 568
  6. 2020Dataset Condensation with Gradient Matchingcited by 510
  7. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  8. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  9. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  10. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  11. 2017Compressed Sensing using Generative Modelscited by 337
  12. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  13. 2017Variational Continual Learningcited by 335
  14. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  15. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  16. 2016Hyperparameter optimization with approximate gradientcited by 231
  17. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  18. 2018Task-Free Continual Learningcited by 219
  19. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  20. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  21. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  22. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
  23. 2019Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Costcited by 114
  24. 2016Coresets for Scalable Bayesian Logistic Regressioncited by 79
  25. 2017Practical Coreset Constructions for Machine Learningcited by 75
  26. 2018Bayesian Coreset Construction via Greedy Iterative Geodesic Ascentcited by 68
  27. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  28. 2017Automated Scalable Bayesian Inference via Hilbert Coresetscited by 55
  29. 2020Introduction to Core-sets: an Updated Surveycited by 45
  30. 2021Instance-Optimal Compressed Sensing via Posterior Samplingcited by 43
  31. 2017Training Gaussian Mixture Models at Scale via Coresetscited by 38
  32. 2018Data Dropout: Optimizing Training Data for Convolutional Neural Networkscited by 23
  33. 2019Combining MixMatch and Active Learning for Better Accuracy with Fewer Labelscited by 17
  34. 2020Semi-supervised Batch Active Learning via Bilevel Optimizationcited by 16
  35. 2013iCub World: Friendly Robots Help Building Good Vision Data-Setscited by 10
  36. 2017Greedy Algorithms for Cone Constrained Optimization with Convergence Guaranteescited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 2.

7 papers.

  1. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  2. 2022Fast Rates in Pool-Based Batch Active Learningcited by 15
  3. 2025Direct Prediction Set Minimization via Bilevel Conformal Classifier Trainingcited by 8
  4. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  5. 2023Soft Random Sampling: A Theoretical and Empirical Analysiscited by 2
  6. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  7. 2023Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimizationcited by 0