# Injectivity of ReLU networks: perspectives from statistical physics

arXiv: https://arxiv.org/abs/2302.14112 · HTML: https://mlsanity.com/paper/2302.14112/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 85 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Compressed Sensing using Generative Models](https://mlsanity.com/paper/1703.03208/) (2017)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Flows for simultaneous manifold learning and density estimation](https://mlsanity.com/paper/2003.13913/) (2020)
- [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017)
- [Exactly Computing the Local Lipschitz Constant of ReLU Networks](https://mlsanity.com/paper/2003.01219/) (2020)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [Universality of empirical risk minimization](https://mlsanity.com/paper/2202.08832/) (2022)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [Trumpets: Injective Flows for Inference and Inverse Problems](https://mlsanity.com/paper/2102.10461/) (2021)
- [Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows](https://mlsanity.com/paper/2106.05275/) (2021)
- [Approximability of Discriminators Implies Diversity in GANs](https://mlsanity.com/paper/1806.10586/) (2018)
- [Another look at the Gardner problem](https://mlsanity.com/paper/1306.3979/) (2013)
- [Globally Injective ReLU Networks](https://mlsanity.com/paper/2006.08464/) (2020)
- [Notes on computational-to-statistical gaps: predictions using statistical physics](https://mlsanity.com/paper/1803.11132/) (2018)
- [Optimization of the Sherrington-Kirkpatrick Hamiltonian](https://mlsanity.com/paper/1812.10897/) (2018)
- [Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows](https://mlsanity.com/paper/2110.04227/) (2021)
- [Tractability from overparametrization: The example of the negative perceptron](https://mlsanity.com/paper/2110.15824/) (2021)
- [Optimization of Mean-field Spin Glasses](https://mlsanity.com/paper/2001.00904/) (2020)
- [Algorithmic Thresholds in Mean Field Spin Glasses](https://mlsanity.com/paper/2009.11481/) (2020)

## Built on by (most cited first, 10 of 10)

- [Bayes-optimal learning of an extensive-width neural network from quadratically many samples](https://mlsanity.com/paper/2408.03733/) (2024)
- [Injectivity capacity of ReLU gates](https://mlsanity.com/paper/2410.20646/) (2024)
- [Deep ReLU networks -- injectivity capacity upper bounds](https://mlsanity.com/paper/2412.19677/) (2024)
- [Spring-block theory of feature learning in deep neural networks](https://mlsanity.com/paper/2407.19353/) (2024)
- [Online Federation For Mixtures of Proprietary Agents with Black-Box Encoders](https://mlsanity.com/paper/2505.00216/) (2025)
- [Factual recall in linear associative memories: sharp asymptotics and mechanistic insights](https://mlsanity.com/paper/2605.10795/) (2026)
- [Injectivity of ReLU-layers: Tools from Frame Theory](https://mlsanity.com/paper/2406.15856/) (2024)
- [Phase transition of descending phase retrieval algorithms](https://mlsanity.com/paper/2506.18275/) (2025)
- [Optimal spectral initializers impact on phase retrieval phase transitions -- an RDT view](https://mlsanity.com/paper/2506.18279/) (2025)
- [Phase retrieval with rank d measurements -- descending algorithms phase transitions](https://mlsanity.com/paper/2506.18282/) (2025)

## Abstract (arXiv)

When can the input of a ReLU neural network be inferred from its output? In other words, when is the network injective? We consider a single layer, x mapsto ReLU(Wx), with a random Gaussian m × n matrix W, in a high-dimensional setting where n, m → ∞. Recent work connects this problem to spherical integral geometry giving rise to a conjectured sharp injectivity threshold for α = fracmn by studying the expected Euler characteristic of a certain random set. We adopt a different perspective and show that injectivity is equivalent to a property of the ground state of the spherical perceptron, an important spin glass model in statistical physics. By leveraging the (non-rigorous) replica symmetry-breaking theory, we derive analytical equations for the threshold whose solution is at odds with that from the Euler characteristic. Furthermore, we use Gordon's min--max theorem to prove that a replica-symmetric upper bound refutes the Euler characteristic prediction. Along the way we aim to give a tutorial-style introduction to key ideas from statistical physics in an effort to make the exposition accessible to a broad audience. Our analysis establishes a connection between spin glasses and integral geometry but leaves open the problem of explaining the discrepancies.