mlsanityData to 14 Oct 2026

Deep neural networks architectures from the perspective of manifold learning

arXiv 2306.03406 ↗First posted 6 Jun 2023cs.LG cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite significant advances in the field of deep learning in ap-plications to various areas, an explanation of the learning pro-cess of neural network models remains an important open ques-tion. The purpose of this paper is a comprehensive comparison and description of neural network architectures in terms of ge-ometry and topology. We focus on the internal representation of neural networks and on the dynamics of changes in the topology and geometry of a data manifold on different layers. In this paper, we use the concepts of topological data analysis (TDA) and persistent homological fractal dimension. We present a wide range of experiments with various datasets and configurations of convolutional neural network (CNNs) architectures and Transformers in CV and NLP tasks. Our work is a contribution to the development of the important field of explainable and interpretable AI within the framework of geometrical deep learning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 86 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

43 of 86 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2017Squeeze-and-Excitation Networkscited by 4,789
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  12. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
31 more
  1. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  2. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  3. 2022Patches Are All You Need?cited by 201
  4. 2017Searching for Activation Functionscited by 169
  5. 2019Ripser: efficient computation of Vietoris-Rips persistence barcodescited by 117
  6. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  7. 2020Topology of deep neural networkscited by 112
  8. 2020CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotationscited by 109
  9. 2018Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topologycited by 73
  10. 2021On the geometry of generalization and memorization in deep neural networkscited by 73
  11. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  12. 2021Scikit-dimension: a Python package for intrinsic dimension estimationcited by 68
  13. 2021Deep Learning for Face Anti-Spoofing: A Surveycited by 65
  14. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  15. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  16. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  17. 2018On the Intrinsic Dimensionality of Image Representationscited by 57
  18. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  19. 2020Computing the Testing Error without a Testing Setcited by 54
  20. 2018Generalization Error in Deep Learningcited by 47
  21. 2020Emergence of Separable Manifolds in Deep Language Representationscited by 40
  22. 2018On Characterizing the Capacity of Neural Networks using Algebraic Topologycited by 38
  23. 2020Empirical Studies on the Properties of Linear Regions in Deep Neural Networkscited by 36
  24. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  25. 2020Hierarchical nucleation in deep neural networkscited by 31
  26. 2018Topological Approaches to Deep Learningcited by 29
  27. 2018Topological Data Analysis of Decision Boundaries with Application to Model Selectioncited by 19
  28. 2021Activation Landscapes as a Topological Summary of Neural Network Performancecited by 18
  29. 2016Efficient Representation of Low-Dimensional Manifolds using Deep Networkscited by 13
  30. 2018Curvature-based Comparison of Two Neural Networkscited by 6
  31. 2019Abstraction Mechanisms Predict Generalization in Deep Neural Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2023AI-generated text boundary detection with RoFTcited by 20
  2. 2024Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Datacited by 5
  3. 2024Riemannian Federated Learning via Averaging Gradient Streamscited by 4
  4. 2023On original and latent space connectivity in deep neural networkscited by 1
  5. 2024Quantifying Manifolds: Do the manifolds learned by Generative Adversarial Networks converge to the real data manifoldcited by 0