mlsanityData to 7 Oct 2026

A Law of Data Separation in Deep Learning

arXiv 2210.17020 ↗First posted 31 Oct 2022cs.LG cs.AI cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

While deep learning has enabled significant advances in many areas of science, its black-box nature hinders architecture design for future artificial intelligence applications and interpretation for high-stakes decision makings. We addressed this issue by studying the fundamental question of how deep neural networks process data in the intermediate layers. Our finding is a simple and quantitative law that governs how deep neural networks separate data according to class membership throughout all layers for classification. This law shows that each layer improves data separation at a constant geometric rate, and its emergence is observed in a collection of network architectures and datasets during training. This law offers practical guidelines for designing architectures, improving model robustness and out-of-sample performance, as well as interpreting the predictions.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

32 of 54 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,279
20 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2016Error bounds for approximations with deep ReLU networkscited by 567
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  9. 2015Training Very Deep Networkscited by 172
  10. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  11. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  12. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  13. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  14. 2019Are All Layers Created Equal?cited by 94
  15. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  16. 2014Convolutional Neural Networks at Constrained Time Costcited by 81
  17. 2019The Local Elasticity of Neural Networkscited by 37
  18. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  19. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  20. 2021Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristicscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 16; 2025: 8; 2026: 3.

36 papers.

Sort:

  1. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  2. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  3. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  4. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  5. 2023Detecting Out-of-Distribution Through the Lens of Neural Collapsecited by 33
  6. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  7. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  8. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  9. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  10. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  11. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  12. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
24 more
  1. 2024A Law of Next-Token Prediction in Large Language Modelscited by 15
  2. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  3. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  4. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  5. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  6. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  7. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  8. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 8
  9. 2024Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learningcited by 8
  10. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  11. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  12. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  13. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  14. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  15. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  16. 2024Collapsed Language Models Promote Fairnesscited by 1
  17. 2025Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimensioncited by 1
  18. 2025Continuous Knowledge-Preserving Decomposition with Adaptive Layer Selection for Few-Shot Class-Incremental Learningcited by 1
  19. 2025Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagationcited by 1
  20. 2026Enhancing classification accuracy through chaoscited by 1
  21. 2026Deep Neural Regression Collapsecited by 1
  22. 2025Are classical deep neural networks weakly adversarially robust?cited by 0
  23. 2025Geometric origin of adversarial vulnerability in deep learningcited by 0
  24. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0