# Hierarchical nucleation in deep neural networks

arXiv: https://arxiv.org/abs/2007.03506 · HTML: https://mlsanity.com/paper/2007.03506/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 30 of 31)

- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Neural networks trained with SGD learn distributions of increasing complexity](https://mlsanity.com/paper/2211.11567/) (2022)
- [Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals](https://mlsanity.com/paper/2402.11655/) (2024)
- [DADApy: Distance-based Analysis of DAta-manifolds in Python](https://mlsanity.com/paper/2205.03373/) (2022)
- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [The representation landscape of few-shot learning and fine-tuning in large language models](https://mlsanity.com/paper/2409.03662/) (2024)
- [Topology and geometry of data manifold in deep learning](https://mlsanity.com/paper/2204.08624/) (2022)
- [Persistent Topological Features in Large Language Models](https://mlsanity.com/paper/2410.11042/) (2024)
- [Data Representations' Study of Latent Image Manifolds](https://mlsanity.com/paper/2305.19730/) (2023)
- [Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers](https://mlsanity.com/paper/2412.06639/) (2024)
- [GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning](https://mlsanity.com/paper/2412.09250/) (2024)
- [How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model](https://mlsanity.com/paper/2404.10727/) (2024)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Deep neural networks architectures from the perspective of manifold learning](https://mlsanity.com/paper/2306.03406/) (2023)
- [Analyzing Deep Transformer Models for Time Series Forecasting via Manifold Learning](https://mlsanity.com/paper/2410.13792/) (2024)
- [The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models](https://mlsanity.com/paper/2412.06646/) (2024)
- [Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks](https://mlsanity.com/paper/2205.13816/) (2022)
- [Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance](https://mlsanity.com/paper/2402.11742/) (2024)
- [Fermi-Bose Machine achieves both generalization and adversarial robustness](https://mlsanity.com/paper/2404.13631/) (2024)
- [What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective](https://mlsanity.com/paper/2206.02454/) (2022)
- [Optimal transfer protocol by incremental layer defrosting](https://mlsanity.com/paper/2303.01429/) (2023)
- [Systematic Architectural Design of Scale Transformed Attention Condenser DNNs via Multi-Scale Class Representational Response Similarity Analysis](https://mlsanity.com/paper/2306.10128/) (2023)
- [Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research](https://mlsanity.com/paper/2403.08438/) (2024)
- [Effects of Common Regularization Techniques on Open-Set Recognition](https://mlsanity.com/paper/2409.12217/) (2024)
- [Geometric origin of adversarial vulnerability in deep learning](https://mlsanity.com/paper/2509.01235/) (2025)
- [Visual Instruction Tuning Aligns Modalities through Abstraction](https://mlsanity.com/paper/2606.03871/) (2026)
- [Task- and dataset-specific information in protein language models](https://mlsanity.com/paper/2608.12090/) (2026)

## Abstract (arXiv)

Deep convolutional networks (DCNs) learn meaningful representations where data that share the same abstract characteristics are positioned closer and closer. Understanding these representations and how they are generated is of unquestioned practical and theoretical interest. In this work we study the evolution of the probability density of the ImageNet dataset across the hidden layers in some state-of-the-art DCNs. We find that the initial layers generate a unimodal probability density getting rid of any structure irrelevant for classification. In subsequent layers density peaks arise in a hierarchical fashion that mirrors the semantic hierarchy of the concepts. Density peaks corresponding to single categories appear only close to the output and via a very sharp transition which resembles the nucleation process of a heterogeneous liquid. This process leaves a footprint in the probability density of the output layer where the topography of the peaks allows reconstructing the semantic relationships of the categories.