# Mutual Information, Neural Networks and the Renormalization Group

arXiv: https://arxiv.org/abs/1704.06279 · HTML: https://mlsanity.com/paper/1704.06279/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2017.

## Built on by (most cited first, 30 of 33)

- [Nonperturbative renormalization for the neural network-QFT correspondence](https://mlsanity.com/paper/2108.01403/) (2021)
- [Tensor networks for interpretable and efficient quantum-inspired machine learning](https://mlsanity.com/paper/2311.11258/) (2023)
- [Renormalizing Diffusion Models](https://mlsanity.com/paper/2308.12355/) (2023)
- [Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group](https://mlsanity.com/paper/2107.06898/) (2021)
- [Tensor networks and efficient descriptions of classical data](https://mlsanity.com/paper/2103.06872/) (2021)
- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [Bayesian RG Flow in Neural Network Field Theories](https://mlsanity.com/paper/2405.17538/) (2024)
- [Universality of Winning Tickets: A Renormalization Group Perspective](https://mlsanity.com/paper/2110.03210/) (2021)
- [Wilsonian Renormalization of Neural Network Gaussian Processes](https://mlsanity.com/paper/2405.06008/) (2024)
- [The Autodidactic Universe](https://mlsanity.com/paper/2104.03902/) (2021)
- [Multilevel Generative Samplers for Investigating Critical Phenomena](https://mlsanity.com/paper/2503.08918/) (2025)
- [Multi-Scale Simulation of Complex Systems: A Perspective of Integrating Knowledge and Data](https://mlsanity.com/paper/2306.10275/) (2023)
- [GUD: Generation with Unified Diffusion](https://mlsanity.com/paper/2410.02667/) (2024)
- [Neural Information Squeezer for Causal Emergence](https://mlsanity.com/paper/2201.10154/) (2022)
- [NFDLM: A Lightweight Network Flow based Deep Learning Model for DDoS Attack Detection in IoT Domains](https://mlsanity.com/paper/2207.10803/) (2022)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [Differentiable Programming of Isometric Tensor Networks](https://mlsanity.com/paper/2110.03898/) (2021)
- [Learning entanglement breakdown as a phase transition by confusion](https://mlsanity.com/paper/2202.00348/) (2022)
- [Three Learning Stages and Accuracy-Efficiency Tradeoff of Restricted Boltzmann Machines](https://mlsanity.com/paper/2209.00873/) (2022)
- [MS3D: A RG Flow-Based Regularization for GAN Training with Limited Data](https://mlsanity.com/paper/2408.11135/) (2024)
- [A Two-Phase Perspective on Deep Learning Dynamics](https://mlsanity.com/paper/2504.12700/) (2025)
- [Categorical Representation Learning and RG flow operators for algorithmic classifiers](https://mlsanity.com/paper/2203.07975/) (2022)
- [Surveying the space of descriptions of a composite system with machine learning](https://mlsanity.com/paper/2411.18579/) (2024)
- [Data augmentation using diffusion models to enhance inverse Ising inference](https://mlsanity.com/paper/2503.10154/) (2025)
- [Siamese Neural Network for Label-Efficient Critical Phenomena Prediction in 3D Percolation Models](https://mlsanity.com/paper/2507.14159/) (2025)
- [Unsupervised Discovery of Intermediate Phase Order in the Frustrated J1-J2 Heisenberg Model via Prometheus Framework](https://mlsanity.com/paper/2602.21468/) (2026)
- [A renormalization-group inspired lattice-based framework for piecewise generalized linear models](https://mlsanity.com/paper/2605.05493/) (2026)
- [The RG-Flow Transformer: Encoding Scale-Free Dynamics in Scarce EEG](https://mlsanity.com/paper/2607.11950/) (2026)
- [Renormalization Group Flow Matching for Scalable Local Generative Modeling](https://mlsanity.com/paper/2608.23696/) (2026)

## Abstract (arXiv)

Physical systems differring in their microscopic details often display strikingly similar behaviour when probed at macroscopic scales. Those universal properties, largely determining their physical characteristics, are revealed by the powerful renormalization group (RG) procedure, which systematically retains "slow" degrees of freedom and integrates out the rest. However, the important degrees of freedom may be difficult to identify. Here we demonstrate a machine learning algorithm capable of identifying the relevant degrees of freedom and executing RG steps iteratively without any prior knowledge about the system. We introduce an artificial neural network based on a model-independent, information-theoretic characterization of a real-space RG procedure, performing this task. We apply the algorithm to classical statistical physics problems in one and two dimensions. We demonstrate RG flow and extract the Ising critical exponent. Our results demonstrate that machine learning techniques can extract abstract physical concepts and consequently become an integral part of theory- and model-building.