mlsanityData to 7 Oct 2026

Hierarchical nucleation in deep neural networks

arXiv 2007.03506 ↗First posted 7 Jul 2020cs.CV cs.LG stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

Deep convolutional networks (DCNs) learn meaningful representations where data that share the same abstract characteristics are positioned closer and closer. Understanding these representations and how they are generated is of unquestioned practical and theoretical interest. In this work we study the evolution of the probability density of the ImageNet dataset across the hidden layers in some state-of-the-art DCNs. We find that the initial layers generate a unimodal probability density getting rid of any structure irrelevant for classification. In subsequent layers density peaks arise in a hierarchical fashion that mirrors the semantic hierarchy of the concepts. Density peaks corresponding to single categories appear only close to the output and via a very sharp transition which resembles the nucleation process of a heterogeneous liquid. This process leaves a footprint in the probability density of the output layer where the topography of the peaks allows reconstructing the semantic relationships of the categories.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 6; 2024: 13; 2025: 3; 2026: 3.

31 papers.

Sort:

  1. 2023The geometry of hidden representations of large transformer modelscited by 108
  2. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  3. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  4. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  5. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  6. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  7. 2022DADApy: Distance-based Analysis of DAta-manifolds in Pythoncited by 22
  8. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  9. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  10. 2022Topology and geometry of data manifold in deep learningcited by 12
  11. 2023Data Representations' Study of Latent Image Manifoldscited by 11
  12. 2024Persistent Topological Features in Large Language Modelscited by 11
19 more
  1. 2024Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformerscited by 11
  2. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  3. 2024GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuningcited by 7
  4. 2025A quantitative analysis of semantic information in deep representations of text and imagescited by 7
  5. 2023Deep neural networks architectures from the perspective of manifold learningcited by 5
  6. 2024Analyzing Deep Transformer Models for Time Series Forecasting via Manifold Learningcited by 5
  7. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Modelscited by 4
  8. 2022Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networkscited by 3
  9. 2024Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalancecited by 2
  10. 2024Fermi-Bose Machine achieves both generalization and adversarial robustnesscited by 2
  11. 2022What do CNNs Learn in the First Layer and Why? A Linear Systems Perspectivecited by 1
  12. 2023Optimal transfer protocol by incremental layer defrostingcited by 0
  13. 2023Systematic Architectural Design of Scale Transformed Attention Condenser DNNs via Multi-Scale Class Representational Response Similarity Analysiscited by 0
  14. 2024Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Researchcited by 0
  15. 2024Effects of Common Regularization Techniques on Open-Set Recognitioncited by 0
  16. 2025Geometric origin of adversarial vulnerability in deep learningcited by 0
  17. 2026Visual Instruction Tuning Aligns Modalities through Abstractioncited by 0
  18. 2026Task- and dataset-specific information in protein language modelscited by 0
  19. 2026Generalization in Neural Networks Through the Lens of Magnitude Potentialcited by 0