mlsanityData to 14 Oct 2026

Can we achieve robustness from data alone?

arXiv 2207.11727 ↗First posted 24 Jul 2022cs.LG cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

We introduce a meta-learning algorithm for adversarially robust classification. The proposed method tries to be as model agnostic as possible and optimizes a dataset prior to its deployment in a machine learning system, aiming to effectively erase its non-robust features. Once the dataset has been created, in principle no specialized algorithm (besides standard gradient descent) is needed to train a robust model. We formulate the data optimization procedure as a bi-level optimization problem on kernel regression, with a class of kernels that describe infinitely wide neural nets (Neural Tangent Kernels). We present extensive experiments on standard computer vision benchmarks using a variety of different models, demonstrating the effectiveness of our method, while also pointing out its current shortcomings. In parallel, we revisit prior work that also focused on the problem of data optimization for robust classification citepIly+19, and show that being robust to adversarial attacks after standard (gradient descent) training on a suitable dataset is more challenging than previously thought.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 24 Jul 2022.

Builds on

49 of 61 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  12. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
37 more
  1. 2016Adversarial examples in the physical worldcited by 1,769
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  4. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  5. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  6. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  7. 2018Robustness May Be at Odds with Accuracycited by 880
  8. 2016Universal adversarial perturbationscited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  10. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  11. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  12. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  13. 2019Adversarial Training for Free!cited by 643
  14. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  15. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  16. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  17. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  18. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  19. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  20. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  21. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  22. 2018Dataset Distillationcited by 230
  23. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  24. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  25. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  26. 2020Training independent subnetworks for robust predictioncited by 177
  27. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  28. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 150
  29. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  30. 2016Technical Report on the CleverHans v2.1.0 Adversarial Examples Librarycited by 124
  31. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  32. 2020Neural Kernels Without Tangentscited by 56
  33. 2021Mind the box: l1-APGD for sparse adversarial attacks on image classifierscited by 51
  34. 2021What can linearized neural networks actually say about generalization?cited by 49
  35. 2022Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networkscited by 21
  36. 2019Adversarially Robust Low Dimensional Representationscited by 7
  37. 2018A Spectral View of Adversarially Robust Featurescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 4; 2025: 1; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 115
  3. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 84
  4. 2024A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensationcited by 57
  5. 2023A Survey on Dataset Distillation: Approaches, Applications and Future Directionscited by 32
  6. 2023Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularizationcited by 21
  7. 2023Adversarial Examples Are Not Real Featurescited by 16
  8. 2023Dataset Condensation via Generative Modelcited by 15
  9. 2024Generative Dataset Distillation Based on Diffusion Modelcited by 13
  10. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 7
  11. 2024Improve Cross-Architecture Generalization on Dataset Distillationcited by 7
  12. 2023Data Optimization in Deep Learning: A Surveycited by 6
4 more
  1. 2023Embarassingly Simple Dataset Distillationcited by 5
  2. 2025A Discrepancy-Based Perspective on Dataset Condensationcited by 1
  3. 2023Kernels, Data & Physicscited by 0
  4. 2026Adversarial Samples Are Not Created Equalcited by 0