mlsanityData to 7 Oct 2026

Understanding Deep Neural Networks via Linear Separability of Hidden Layers

arXiv 2307.13962 ↗First posted 26 Jul 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we measure the linear separability of hidden layer outputs to study the characteristics of deep neural networks. In particular, we first propose Minkowski difference based linear separability measures (MD-LSMs) to evaluate the linear separability degree of two points sets. Then, we demonstrate that there is a synchronicity between the linear separability degree of hidden layer outputs and the network training performance, i.e., if the updated weights can enhance the linear separability degree of hidden layer outputs, the updated network will achieve a better training performance, and vice versa. Moreover, we study the effect of activation function and network size (including width and depth) on the linear separability of hidden layers. Finally, we conduct the numerical experiments to validate our findings on some popular deep networks including multilayer perceptron (MLP), convolutional neural network (CNN), deep belief network (DBN), ResNet, VGGNet, AlexNet, vision transformer (ViT) and GoogLeNet.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

23 of 51 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
11 more
  1. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  2. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  3. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  4. 2020Causal Intervention for Weakly-Supervised Semantic Segmentationcited by 229
  5. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  6. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77
  7. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  8. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  9. 2020Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attentioncited by 45
  10. 2021The Emergence of Objectness: Learning Zero-Shot Segmentation from Videoscited by 45
  11. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Embedded Visual Prompt Tuningcited by 7
  2. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  3. 2025Geometric Properties and Graph-Based Optimization of Neural Networks: Addressing Non-Linearity, Dimensionality, and Scalabilitycited by 0
  4. 2026Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Studycited by 0