# Absolute abstraction: a renormalisation group approach

arXiv: https://arxiv.org/abs/2407.01656 · HTML: https://mlsanity.com/paper/2407.01656/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 84 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [The staircase property: How hierarchical structure can guide deep learning](https://mlsanity.com/paper/2108.10573/) (2021)
- [An exact mapping between the Variational Renormalization Group and Deep Learning](https://mlsanity.com/paper/1410.3831/) (2014)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [Data-driven emergence of convolutional structure in neural networks](https://mlsanity.com/paper/2202.00565/) (2022)
- [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021)
- [Mean-field inference methods for neural networks](https://mlsanity.com/paper/1911.00890/) (2019)
- [Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow](https://mlsanity.com/paper/1801.07172/) (2018)
- [Restricted Boltzmann Machines: Introduction and Review](https://mlsanity.com/paper/1806.07066/) (2018)
- [Explaining the effects of non-convergent sampling in the training of Energy-Based Models](https://mlsanity.com/paper/2301.09428/) (2023)
- [Self-Organized Criticality in the Brain](https://mlsanity.com/paper/2102.09124/) (2021)
- [Quantifying Relevance in Learning and Inference](https://mlsanity.com/paper/2202.00339/) (2022)
- [Statistical models of neural activity, criticality, and Zipf's law](https://mlsanity.com/paper/1812.09123/) (2018)
- [A random energy approach to deep learning](https://mlsanity.com/paper/2112.09420/) (2021)
- [A simple probabilistic neural network for machine understanding](https://mlsanity.com/paper/2210.13179/) (2022)

## Abstract (arXiv)

Abstraction is the process of extracting the essential features from raw data while ignoring irrelevant details. It is well known that abstraction emerges with depth in neural networks, where deep layers capture abstract characteristics of data by combining lower level features encoded in shallow layers (e.g. edges). Yet we argue that depth alone is not enough to develop truly abstract representations. We advocate that the level of abstraction crucially depends on how broad the training set is. We address the issue within a renormalisation group approach where a representation is expanded to encompass a broader set of data. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for a representation which is absolutely abstract. This theoretical picture is tested in numerical experiments based on Deep Belief Networks and auto-encoders trained on data of different breadth. These show that representations in neural networks approach the Hierarchical Feature Model as the data get broader and as depth increases, in agreement with theoretical predictions.