# Exploring the Function Space of Deep-Learning Machines

arXiv: https://arxiv.org/abs/1708.01422 · HTML: https://mlsanity.com/paper/1708.01422/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2017.

## Built on by (most cited first, 7 of 7)

- [Exploring Loss Landscapes through the Lens of Spin Glass Theory](https://mlsanity.com/paper/2407.20724/) (2024)
- [A random energy approach to deep learning](https://mlsanity.com/paper/2112.09420/) (2021)
- [Initializing ReLU networks in an expressive subspace of weights](https://mlsanity.com/paper/2103.12499/) (2021)
- [Bayesian Deep Learning Hyperparameter Search for Robust Function Mapping to Polynomials with Noise](https://mlsanity.com/paper/2106.12532/) (2021)
- [Dynamics of Meta-learning Representation in the Teacher-student Scenario](https://mlsanity.com/paper/2408.12545/) (2024)
- [Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?](https://mlsanity.com/paper/2511.07308/) (2025)
- [Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks](https://mlsanity.com/paper/2608.06597/) (2026)

## Abstract (arXiv)

The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely-connected architectures to discover a layer-wise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.