# Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks

arXiv: https://arxiv.org/abs/2305.05089 · HTML: https://mlsanity.com/paper/2305.05089/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 12 of 12)

- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets](https://mlsanity.com/paper/1906.06247/) (2019)
- [Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape](https://mlsanity.com/paper/1907.02911/) (2019)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [Semi-flat minima and saddle points by embedding neural networks to overparameterization](https://mlsanity.com/paper/1906.04868/) (2019)
- [Parameter identifiability of a deep feedforward ReLU neural network](https://mlsanity.com/paper/2112.12982/) (2021)
- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [Affine symmetries and neural network identifiability](https://mlsanity.com/paper/2006.11727/) (2020)
- [Neural network identifiability for a family of sigmoidal nonlinearities](https://mlsanity.com/paper/1906.06994/) (2019)
- [Frivolous Units: Wider Networks Are Not Really That Wide](https://mlsanity.com/paper/1912.04783/) (2019)

## Built on by (most cited first, 6 of 6)

- [The Local Learning Coefficient: A Singularity-Aware Complexity Measure](https://mlsanity.com/paper/2308.12108/) (2023)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Estimating the Local Learning Coefficient at Scale](https://mlsanity.com/paper/2402.03698/) (2024)
- [Dead Directions: Geometric Singular Learning](https://mlsanity.com/paper/2606.05957/) (2026)
- [Proximity to Losslessly Compressible Parameters](https://mlsanity.com/paper/2306.02834/) (2023)
- [Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment](https://mlsanity.com/paper/2607.00603/) (2026)

## Abstract (arXiv)

Understanding the learning process of artificial neural networks requires clarifying the structure of the parameter space within which learning takes place. A neural network parameter's functional equivalence class is the set of parameters implementing the same input--output function. For many architectures, almost all parameters have a simple and well-documented functional equivalence class. However, there is also a vanishing minority of reducible parameters, with richer functional equivalence classes caused by redundancies among the network's units. In this paper, we give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture. We show that such functional equivalence classes are piecewise-linear path-connected sets, and that for parameters with a majority of redundant units, the sets have a diameter of at most 7 linear segments.