# An entropy formula for the Deep Linear Network

arXiv: https://arxiv.org/abs/2509.09088 · HTML: https://mlsanity.com/paper/2509.09088/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Directional convergence and alignment in deep learning](https://mlsanity.com/paper/2006.06657/) (2020)
- [No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis](https://mlsanity.com/paper/1704.00708/) (2017)
- [On the Implicit Bias in Deep-Learning Algorithms](https://mlsanity.com/paper/2208.12591/) (2022)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [Geometry of Linear Convolutional Networks](https://mlsanity.com/paper/2108.01538/) (2021)
- [Convergence of gradient descent for learning linear neural networks](https://mlsanity.com/paper/2108.02040/) (2021)
- [Infinite-width limit of deep linear neural networks](https://mlsanity.com/paper/2211.16980/) (2022)
- [Function Space and Critical Points of Linear Convolutional Networks](https://mlsanity.com/paper/2304.05752/) (2023)
- [The geometry of the deep linear network](https://mlsanity.com/paper/2411.09004/) (2024)
- [Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit](https://mlsanity.com/paper/2210.12497/) (2022)
- [Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss](https://mlsanity.com/paper/2303.03027/) (2023)
- [Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizers](https://mlsanity.com/paper/2309.10639/) (2023)
- [Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensing](https://mlsanity.com/paper/2402.17595/) (2024)

## Built on by (most cited first, 8 of 8)

- [Regularization Implies balancedness in the deep linear network](https://mlsanity.com/paper/2511.01137/) (2025)
- [Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective](https://mlsanity.com/paper/2601.06597/) (2026)
- [On the implicit regularization of Langevin dynamics with projected noise](https://mlsanity.com/paper/2602.12257/) (2026)
- [Geometric and Spectral Alignment for Deep Neural Network I](https://mlsanity.com/paper/2605.02108/) (2026)
- [Geodesics in the Deep Linear Network](https://mlsanity.com/paper/2510.07324/) (2025)
- [On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks](https://mlsanity.com/paper/2606.16579/) (2026)
- [Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks](https://mlsanity.com/paper/2608.06597/) (2026)
- [Special Lagrangian cones in Deep Learning](https://mlsanity.com/paper/2609.20159/) (2026)

## Abstract (arXiv)

We study the Riemannian geometry of the Deep Linear Network (DLN) as a foundation for a thermodynamic description of the learning process. The main tools are the use of group actions to analyze overparametrization and the use of Riemannian submersion from the space of parameters to the space of observables. The foliation of the balanced manifold in the parameter space by group orbits is used to define and compute a Boltzmann entropy. We also show that the Riemannian geometry on the space of observables defined in [2] is obtained by Riemannian submersion of the balanced manifold. The main technical step is an explicit construction of an orthonormal basis for the tangent space of the balanced manifold using the theory of Jacobi matrices.