# Globally Injective ReLU Networks

arXiv: https://arxiv.org/abs/2006.08464 · HTML: https://mlsanity.com/paper/2006.08464/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 23 of 23)

- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Universal Approximation Theorems for Differentiable Geometric Deep Learning](https://mlsanity.com/paper/2101.05390/) (2021)
- [Transporting Causal Mechanisms for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2107.11055/) (2021)
- [Trumpets: Injective Flows for Inference and Inverse Problems](https://mlsanity.com/paper/2102.10461/) (2021)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values](https://mlsanity.com/paper/2203.01993/) (2022)
- [SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer](https://mlsanity.com/paper/2301.12811/) (2023)
- [Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows](https://mlsanity.com/paper/2110.04227/) (2021)
- [Conditional Injective Flows for Bayesian Imaging](https://mlsanity.com/paper/2204.07664/) (2022)
- [Globally injective and bijective neural operators](https://mlsanity.com/paper/2306.03982/) (2023)
- [Small Transformers Compute Universal Metric Embeddings](https://mlsanity.com/paper/2209.06788/) (2022)
- [Injectivity of ReLU networks: perspectives from statistical physics](https://mlsanity.com/paper/2302.14112/) (2023)
- [Differentiable Generalized Sliced Wasserstein Plans](https://mlsanity.com/paper/2505.22049/) (2025)
- [Continuous Generative Neural Networks: A Wavelet-Based Architecture in Function Spaces](https://mlsanity.com/paper/2205.14627/) (2022)
- [Injectivity capacity of ReLU gates](https://mlsanity.com/paper/2410.20646/) (2024)
- [On the Relationship Between Robustness and Expressivity of Graph Neural Networks](https://mlsanity.com/paper/2504.13786/) (2025)
- [Convex Geometry of ReLU-layers, Injectivity on the Ball and Local Reconstruction](https://mlsanity.com/paper/2307.09672/) (2023)
- [Complexity of Injectivity and Verification of ReLU Neural Networks](https://mlsanity.com/paper/2405.19805/) (2024)
- [Can neural operators always be continuously discretized?](https://mlsanity.com/paper/2412.03393/) (2024)
- [ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers](https://mlsanity.com/paper/2505.20032/) (2025)
- [Exact Feature Collisions in Neural Networks](https://mlsanity.com/paper/2205.15763/) (2022)
- [Injectivity of ReLU-layers: Tools from Frame Theory](https://mlsanity.com/paper/2406.15856/) (2024)
- [A Unifying Relational Perspective on Expressive Lottery Tickets](https://mlsanity.com/paper/2608.18819/) (2026)

## Abstract (arXiv)

Injectivity plays an important role in generative models where it enables inference; in inverse problems and compressed sensing with generative priors it is a precursor to well posedness. We establish sharp characterizations of injectivity of fully-connected and convolutional ReLU layers and networks. First, through a layerwise analysis, we show that an expansivity factor of two is necessary and sufficient for injectivity by constructing appropriate weight matrices. We show that global injectivity with iid Gaussian matrices, a commonly used tractable model, requires larger expansivity between 3.4 and 10.5. We also characterize the stability of inverting an injective network via worst-case Lipschitz constants of the inverse. We then use arguments from differential topology to study injectivity of deep networks and prove that any Lipschitz map can be approximated by an injective ReLU network. Finally, using an argument based on random projections, we show that an end-to-end -- rather than layerwise -- doubling of the dimension suffices for injectivity. Our results establish a theoretical basis for the study of nonlinear inverse and inference problems using neural networks.