# Deep Neural Nets with Fixed Bias Configuration

arXiv: https://arxiv.org/abs/2107.01308 · HTML: https://mlsanity.com/paper/2107.01308/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2021.

## Built on by (most cited first, 4 of 4)

- [An Optimal Time Variable Learning Framework for Deep Neural Networks](https://mlsanity.com/paper/2204.08528/) (2022)
- [Lecture Notes: Neural Network Architectures](https://mlsanity.com/paper/2304.05133/) (2023)
- [A Note on Dimensionality Reduction in Deep Neural Networks using Empirical Interpolation Method](https://mlsanity.com/paper/2305.09842/) (2023)
- [Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations](https://mlsanity.com/paper/2604.07715/) (2026)

## Abstract (arXiv)

For any given neural network architecture a permutation of weights and biases results in the same functional network. This implies that optimization algorithms used to `train' or `learn' the network are faced with a very large number (in the millions even for small networks) of equivalent optimal solutions in the parameter space. To the best of our knowledge, this observation is absent in the literature. In order to narrow down the parameter search space, a novel technique is introduced in order to fix the bias vector configurations to be monotonically increasing. This is achieved by augmenting a typical learning problem with inequality constraints on the bias vectors in each layer. A Moreau-Yosida regularization based algorithm is proposed to handle these inequality constraints and a theoretical convergence of the this algorithm is established. Applications of the proposed approach to standard trigonometric functions and more challenging stiff ordinary differential equations arising in chemically reacting ows clearly illustrate the benefits of the proposed approach.