# The emergence of a concept in shallow neural networks

arXiv: https://arxiv.org/abs/2109.00454 · HTML: https://mlsanity.com/paper/2109.00454/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 39 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors](https://mlsanity.com/paper/1702.05882/) (2017)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Emergence of Compositional Representations in Restricted Boltzmann Machines](https://mlsanity.com/paper/1611.06759/) (2016)
- [Modelling the influence of data structure on learning in neural networks: the hidden manifold model](https://mlsanity.com/paper/1909.11500/) (2019)
- [Neural networks with redundant representation: detecting the undetectable](https://mlsanity.com/paper/1911.12689/) (2019)
- [Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses](https://mlsanity.com/paper/1612.01717/) (2016)
- [Statistical physics of unsupervised learning with prior knowledge in neural networks](https://mlsanity.com/paper/1911.02344/) (2019)

## Built on by (most cited first, 20 of 20)

- [Supervised Hebbian Learning](https://mlsanity.com/paper/2203.01304/) (2022)
- [Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning](https://mlsanity.com/paper/2211.14067/) (2022)
- [Dense Hebbian neural networks: a replica symmetric picture of supervised learning](https://mlsanity.com/paper/2212.00606/) (2022)
- [Random Features Hopfield Networks generalize retrieval to previously unseen examples](https://mlsanity.com/paper/2407.05658/) (2024)
- [Hopfield model with planted patterns: a teacher-student self-supervised learning model](https://mlsanity.com/paper/2304.13710/) (2023)
- [Efficient shallow learning as an alternative to deep learning](https://mlsanity.com/paper/2211.11106/) (2022)
- [Low-latency vision transformers via large-scale multi-head attention](https://mlsanity.com/paper/2506.23832/) (2025)
- [Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting](https://mlsanity.com/paper/2308.01421/) (2023)
- [Parallel Learning by Multitasking Neural Networks](https://mlsanity.com/paper/2308.04106/) (2023)
- [Hebbian Learning from First Principles](https://mlsanity.com/paper/2401.07110/) (2024)
- [Recurrent neural networks that generalize from examples and optimize by dreaming](https://mlsanity.com/paper/2204.07954/) (2022)
- [Statistical Mechanics of Learning via Reverberation in Bidirectional Associative Memories](https://mlsanity.com/paper/2307.08365/) (2023)
- [Clinical Domain Knowledge-Derived Template Improves Post Hoc AI Explanations in Pneumothorax Classification](https://mlsanity.com/paper/2403.18871/) (2024)
- [Role of Delay in Brain Dynamics](https://mlsanity.com/paper/2410.11384/) (2024)
- [Neural Learning Rules from Associative Networks Theory](https://mlsanity.com/paper/2503.19922/) (2025)
- [Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines](https://mlsanity.com/paper/2209.06932/) (2022)
- [Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking](https://mlsanity.com/paper/2312.09638/) (2023)
- [Geometric origin of adversarial vulnerability in deep learning](https://mlsanity.com/paper/2509.01235/) (2025)
- [Deep Learning Approaches with Explainable AI for Differentiating Alzheimer Disease and Mild Cognitive Impairment](https://mlsanity.com/paper/2510.00048/) (2025)
- [Hierarchical Prototype Emergence in Modern Hopfield Models](https://mlsanity.com/paper/2609.12079/) (2026)

## Abstract (arXiv)

We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable ``archetypes'' and we show that there exists a critical sample size beyond which the RBM can learn archetypes, namely the machine can successfully play as a generative model or as a classifier, according to the operational routine. In general, assessing a critical sample size (possibly in relation to the quality of the dataset) is still an open problem in machine learning. Here, restricting to the random theory, where shallow networks suffice and the grand-mother cell scenario is correct, we leverage the formal equivalence between RBMs and Hopfield networks, to obtain a phase diagram for both the neural architectures which highlights regions, in the space of the control parameters (i.e., number of archetypes, number of neurons, size and quality of the training set), where learning can be accomplished. Our investigations are led by analytical methods based on the statistical-mechanics of disordered systems and results are further corroborated by extensive Monte Carlo simulations.