# Affine symmetries and neural network identifiability

arXiv: https://arxiv.org/abs/2006.11727 · HTML: https://mlsanity.com/paper/2006.11727/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2020.

## Built on by (most cited first, 18 of 18)

- [Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry](https://mlsanity.com/paper/2304.02902/) (2023)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samples](https://mlsanity.com/paper/2101.07150/) (2021)
- [Identifiable Equivariant Networks are Layerwise Equivariant](https://mlsanity.com/paper/2601.21645/) (2026)
- [Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks](https://mlsanity.com/paper/2305.05089/) (2023)
- [Most ReLU Networks Admit Identifiable Parameters](https://mlsanity.com/paper/2605.03601/) (2026)
- [Finite Sample Identification of Wide Shallow Neural Networks with Biases](https://mlsanity.com/paper/2211.04589/) (2022)
- [Metric Entropy Limits on Recurrent Neural Network Learning of Linear Dynamical Systems](https://mlsanity.com/paper/2105.02556/) (2021)
- [Constraining the outputs of ReLU neural networks](https://mlsanity.com/paper/2508.03867/) (2025)
- [The Symmetries of Three-Layer ReLU Networks](https://mlsanity.com/paper/2605.18319/) (2026)
- [Complete Identification of Deep ReLU Networks through Lukasiewicz Logic](https://mlsanity.com/paper/2602.00266/) (2026)
- [Proximity to Losslessly Compressible Parameters](https://mlsanity.com/paper/2306.02834/) (2023)
- [Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks](https://mlsanity.com/paper/2409.19138/) (2024)
- [Finite Samples for Shallow Neural Networks](https://mlsanity.com/paper/2503.12744/) (2025)
- [Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks](https://mlsanity.com/paper/2608.27113/) (2026)
- [On Parameter Symmetries and Conservation Laws in Gradient Flow](https://mlsanity.com/paper/2609.34549/) (2026)
- [Singular parameters and missing limits in neural PDE solvers](https://mlsanity.com/paper/2610.06770/) (2026)
- [The Identifiability and Observability of Deep Normalized Attention](https://mlsanity.com/paper/2610.09620/) (2026)

## Abstract (arXiv)

We address the following question of neural network identifiability: Suppose we are given a function f:ℝm→ℝn and a nonlinearity ρ. Can we specify the architecture, weights, and biases of all feed-forward neural networks with respect to ρ giving rise to f? Existing literature on the subject suggests that the answer should be yes, provided we are only concerned with finding networks that satisfy certain "genericity conditions". Moreover, the identified networks are mutually related by symmetries of the nonlinearity. For instance, the tanh function is odd, and so flipping the signs of the incoming and outgoing weights of a neuron does not change the output map of the network. The results known hitherto, however, apply either to single-layer networks, or to networks satisfying specific structural assumptions (such as full connectivity), as well as to specific nonlinearities. In an effort to answer the identifiability question in greater generality, we consider arbitrary nonlinearities with potentially complicated affine symmetries, and we show that the symmetries can be used to find a rich set of networks giving rise to the same function f. The set obtained in this manner is, in fact, exhaustive (i.e., it contains all networks giving rise to f) unless there exists a network A "with no internal symmetries" giving rise to the identically zero function. This result can thus be interpreted as an analog of the rank-nullity theorem for linear operators. We furthermore exhibit a class of "tanh-type" nonlinearities (including the tanh function itself) for which such a network A does not exist, thereby solving the identifiability question for these nonlinearities in full generality. Finally, we show that this class contains nonlinearities with arbitrarily complicated symmetries.