mlsanityData to 7 Oct 2026

Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis

arXiv 2410.00396 ↗First posted 1 Oct 2024cond-mat.stat-mech cond-mat.dis-nn cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Deep neural network architectures often consist of repetitive structural elements. We introduce an approach that reveals these patterns and can be broadly applied to the study of deep learning. Similarly to how a power strip helps untangle and organize complex cable connections, this approach treats neurons as additional degrees of freedom in interactions, simplifying the structure and enhancing the intuitive understanding of interactions within deep neural networks. Furthermore, it reveals the translational symmetry of deep neural networks, which simplifies the application of the renormalization group transformation-a method that effectively analyzes the scaling behavior of the system. By utilizing translational symmetry and renormalization group transformations, we can analyze critical phenomena. This approach may open new avenues for studying deep neural networks using statistical physics.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 71 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

28 of 71 references.

References · 28

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2016An overview of gradient descent optimization algorithmscited by 1,021
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2021Explaining Neural Scaling Lawscited by 353
  7. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  8. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  10. 2014The Loss Surfaces of Multilayer Networkscited by 278
  11. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  12. 2019Universal Approximation with Deep Narrow Networkscited by 198
16 more
  1. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  2. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 120
  3. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  4. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  5. 2016Why does deep and cheap learning work so well?cited by 100
  6. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  7. 2020Minimum Width for Universal Approximationcited by 92
  8. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  9. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  10. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 52
  11. 2018Deep, Skinny Neural Networks are not Universal Approximatorscited by 37
  12. 2013Statistical mechanics of complex neural systems and high dimensional datacited by 15
  13. 2018Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flowcited by 14
  14. 2018Restricted Boltzmann Machines: Introduction and Reviewcited by 14
  15. 2018Thermodynamics and Feature Extraction by Machine Learningcited by 7
  16. 2024Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?cited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Bulk-boundary decomposition of neural networkscited by 0