mlsanityData to 7 Oct 2026

Relative stability toward diffeomorphisms indicates performance in deep nets

arXiv 2105.02468 ↗First posted 6 May 2021cs.LG cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Understanding why deep nets can classify data in large dimensions remains a challenge. It has been proposed that they do so by becoming stable to diffeomorphisms, yet existing empirical measurements support that it is often not the case. We revisit this question by defining a maximum-entropy distribution on diffeomorphisms, that allows to study typical diffeomorphisms of a given norm. We confirm that stability toward diffeomorphisms does not strongly correlate to performance on benchmark data sets of images. By contrast, we find that the stability toward diffeomorphisms relative to that of generic transformations Rf correlates remarkably with the test error εt. It is of order unity at initialization but decreases by several decades during training for state-of-the-art architectures. For CIFAR10 and 15 known architectures, we find εt≈ 0.2√Rf, suggesting that obtaining a small Rf is important to achieve good performance. We study how Rf depends on the size of the training set and compare it to a simple model of invariant learning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

47 of 78 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2018On Lazy Training in Differentiable Programmingcited by 671
  10. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
35 more
  1. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  6. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  7. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  8. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  9. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  10. 2018Spatially Transformed Adversarial Examplescited by 195
  11. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  12. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  13. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  14. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  15. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  16. 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 141
  17. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  18. 2016Exploiting Cyclic Symmetry in Convolutional Neural Networkscited by 91
  19. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  20. 2016Understanding Deep Convolutional Networkscited by 76
  21. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  22. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  23. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  24. 2017Geometric robustness of deep networks: analysis and improvementcited by 45
  25. 2018ADef: an Iterative Algorithm to Construct Adversarial Deformationscited by 38
  26. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
  27. 2015An Empirical Evaluation of Deep Learning on Highway Drivingcited by 35
  28. 2018On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functionscited by 31
  29. 2020Geometric compression of invariant manifolds in neural netscited by 28
  30. 2015Manitest: Are classifiers really invariant?cited by 27
  31. 2015Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentationcited by 24
  32. 2020Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Trainingcited by 11
  33. 2020Anatomical Data Augmentation via Fluid-based Image Registrationcited by 9
  34. 2018Pooling is neither necessary nor sufficient for appropriate deformation stability in CNNscited by 8
  35. 2019Neural Spectrum Alignment: Empirical Studycited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 1; 2024: 2.

12 papers.

  1. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  2. 2021PRIME: A few primitives can boost robustness to common corruptionscited by 40
  3. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  4. 2022Automatic Data Augmentation via Invariance-Constrained Learningcited by 17
  5. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  6. 2021On the Sample Complexity of Learning under Invariance and Geometric Stabilitycited by 13
  7. 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
  8. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  9. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  10. 2022How deep convolutional neural networks lose spatial information with trainingcited by 5
  11. 2022Data augmentation with mixtures of max-entropy transformations for filling-level classificationcited by 1
  12. 2022Measuring dissimilarity with diffeomorphism invariancecited by 0