mlsanityData to 7 Oct 2026

Absolute abstraction: a renormalisation group approach

arXiv 2407.01656 ↗First posted 1 Jul 2024cs.LG cond-mat.dis-nn physics.data-ancited by 0

Not on the map. Why?

Abstract (arXiv)

Abstraction is the process of extracting the essential features from raw data while ignoring irrelevant details. It is well known that abstraction emerges with depth in neural networks, where deep layers capture abstract characteristics of data by combining lower level features encoded in shallow layers (e.g. edges). Yet we argue that depth alone is not enough to develop truly abstract representations. We advocate that the level of abstraction crucially depends on how broad the training set is. We address the issue within a renormalisation group approach where a representation is expanded to encompass a broader set of data. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for a representation which is absolutely abstract. This theoretical picture is tested in numerical experiments based on Deep Belief Networks and auto-encoders trained on data of different breadth. These show that representations in neural networks approach the Hierarchical Feature Model as the data get broader and as depth increases, in agreement with theoretical predictions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 84 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

24 of 84 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  7. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  8. 2016Deep Information Propagationcited by 194
  9. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  10. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  11. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  12. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
12 more
  1. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 52
  2. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  3. 2021Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machinescited by 27
  4. 2019Mean-field inference methods for neural networkscited by 16
  5. 2018Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flowcited by 14
  6. 2018Restricted Boltzmann Machines: Introduction and Reviewcited by 14
  7. 2023Explaining the effects of non-convergent sampling in the training of Energy-Based Modelscited by 11
  8. 2021Self-Organized Criticality in the Braincited by 7
  9. 2022Quantifying Relevance in Learning and Inferencecited by 5
  10. 2018Statistical models of neural activity, criticality, and Zipf's lawcited by 2
  11. 2021A random energy approach to deep learningcited by 2
  12. 2022A simple probabilistic neural network for machine understandingcited by 1

Built on by

Not cited yet.