mlsanityData to 7 Oct 2026

Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation

arXiv 2303.05161 ↗First posted 9 Mar 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

To achieve near-zero training error in a classification problem, the layers of a feed-forward network have to disentangle the manifolds of data points with different labels, to facilitate the discrimination. However, excessive class separation can bring to overfitting since good generalisation requires learning invariant features, which involve some level of entanglement. We report on numerical experiments showing how the optimisation dynamics finds representations that balance these opposing tendencies with a non-monotonic trend. After a fast segregation phase, a slower rearrangement (conserved across data sets and architectures) increases the class entanglement.The training error at the inversion is stable under subsampling, and across network initialisations and optimisers, which characterises it as a property solely of the data structure and (very weakly) of the architecture. The inversion is the manifestation of tradeoffs elicited by well-defined and maximally stable elements of the training set, coined ``stragglers'', particularly influential for generalisation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 31 of its 67 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

31 of 67 references.

References · 31

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2020Supervised Contrastive Learningcited by 2,803
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2017A Closer Look at Memorization in Deep Networkscited by 936
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  10. 2014Deep metric learning using Triplet networkcited by 421
  11. 2018Deep Learning for Classical Japanese Literaturecited by 415
  12. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
19 more
  1. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  2. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  3. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  4. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  5. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  6. 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
  7. 2013Deep Learning of Representations: Looking Forwardcited by 84
  8. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  9. 2019Where is the Information in a Deep Neural Network?cited by 40
  10. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
  11. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  12. 2018How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networkscited by 28
  13. 2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 24
  14. 2021Learning through atypical "phase transitions" in overparameterized neural networkscited by 21
  15. 2018Semi-Supervised Learning via Compact Latent Space Clusteringcited by 15
  16. 2022The activity-weight duality in feed forward neural networks: The geometric determinants of generalizationcited by 12
  17. 2019Intrinsic dimension estimation for locally undersampled datacited by 10
  18. 2022Linear Classification of Neural Manifolds with Correlated Variabilitycited by 9
  19. 2020Beyond the storage capacity: data driven satisfiability transitioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 2.

6 papers.

  1. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  2. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  3. 2026Reasoning emerges from constrained inference manifolds in large language modelscited by 5
  4. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  5. 2025The Advantage of Fine-Grained Trainingcited by 1
  6. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1