mlsanityData to 7 Oct 2026

Injectivity capacity of ReLU gates

arXiv 2410.20646 ↗First posted 28 Oct 2024stat.ML cond-mat.dis-nn cs.ITcited by 4

Not on the map. Why?

Abstract (arXiv)

We consider the injectivity property of the ReLU networks layers. Determining the ReLU injectivity capacity (ratio of the number of layer's inputs and outputs) is established as isomorphic to determining the capacity of the so-called ℓ0 spherical perceptron. Employing fully lifted random duality theory (fl RDT) a powerful program is developed and utilized to handle the ℓ0 spherical perceptron and implicitly the ReLU layers injectivity. To put the entire fl RDT machinery in practical use, a sizeable set of numerical evaluations is conducted as well. The lifting mechanism is observed to converge remarkably fast with relative corrections in the estimated quantities not exceeding ~ 0.1% already on the third level of lifting. Closed form explicit analytical relations among key lifting parameters are uncovered as well. In addition to being of incredible importance in handling all the required numerical work, these relations also shed a new light on beautiful parametric interconnections within the lifting structure. Finally, the obtained results are also shown to fairly closely match the replica predictions from [40].

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 78 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

36 of 78 references.

References · 36

  1. 2017Compressed Sensing using Generative Modelscited by 337
  2. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  3. 2016The Little Engine that Could: Regularization by Denoising (RED)cited by 303
  4. 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
  5. 2017A Random Matrix Approach to Neural Networkscited by 80
  6. 2018Phase Retrieval Under a Generative Priorcited by 73
  7. 2018Solving Linear Inverse Problems Using GAN Priors: An Algorithm with Provable Guaranteescited by 58
  8. 2018Neural Proximal Gradient Descent for Compressive Imagingcited by 46
  9. 2019Deep Compressed Sensingcited by 46
  10. 2018Modeling Sparse Deviations for Compressed Sensing using Generative Modelscited by 33
  11. 2019Inverting Deep Generative models, One layer at a timecited by 33
  12. 2021Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flowscited by 30
24 more
  1. 2013Regularly random dualitycited by 23
  2. 2013Another look at the Gardner problemcited by 23
  3. 2020Globally Injective ReLU Networkscited by 23
  4. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
  5. 2020Discrepancy Minimization via a Self-Balancing Walkcited by 22
  6. 2021Proof of the Contiguity Conjecture and Lognormal Limit for the Symmetric Perceptroncited by 18
  7. 2023Fully lifted random duality theorycited by 17
  8. 2013Discrete perceptronscited by 14
  9. 2013Lifting ℓ1-optimization strong and sectional thresholdscited by 13
  10. 2021Universal Joint Approximation of Manifolds and Densities by Simple Injective Flowscited by 13
  11. 2021Binary perceptron: efficient algorithms can find solutions in a rare well-connected clustercited by 13
  12. 2023Globally injective and bijective neural operatorscited by 13
  13. 2023Binary perceptrons capacity via fully lifted random duality theorycited by 13
  14. 2013Signal Recovery from Pooling Representationscited by 12
  15. 2017Inference in Deep Networks in High Dimensionscited by 12
  16. 2022Algorithms and Barriers in the Symmetric Binary Perceptron Modelcited by 12
  17. 2013Spherical perceptron as a storage memory with limited errorscited by 11
  18. 2023Fl RDT based ultimate lowering of the negative spherical perceptron capacitycited by 11
  19. 2023Injectivity of ReLU networks: perspectives from statistical physicscited by 10
  20. 2020Constant-Expansion Suffices for Compressed Sensing with Generative Priorscited by 9
  21. 2013Meshes that trap random subspacescited by 8
  22. 2024Exact capacity of the wide hidden layer treelike neural networks with generic activationscited by 6
  23. 2024Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networkscited by 6
  24. 2024Fixed width treelike neural networks capacity analysis -- generic activationscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Deep ReLU networks -- injectivity capacity upper boundscited by 3
  2. 2025Phase transition of descending phase retrieval algorithmscited by 0
  3. 2025Optimal spectral initializers impact on phase retrieval phase transitions -- an RDT viewcited by 0
  4. 2025Phase retrieval with rank d measurements -- descending algorithms phase transitionscited by 0