Stably unactivated neurons in ReLU neural networks

The choice of architecture of a neural network influences which functions will be realizable by that neural network and, as a result, studying the expressiveness of a chosen architecture has received much attention. In ReLU neural networks, the presence of stably unactivated neurons can reduce the network's expressiveness. In this work, we investigate the probability of a neuron in the second hidden layer of such neural networks being stably unactivated when the weights and biases are initialized from symmetric probability distributions. For networks with input dimension n0, we prove that if the first hidden layer has n0+1 neurons then this probability is exactly frac2n0+14n0+1, and if the first hidden layer has n1 neurons, n1 ≤ n0, then the probability is frac12n1+1. Finally, for the case when the first hidden layer has more neurons than n0+1, a conjecture is proposed along with the rationale. Computational evidence is presented to support the conjecture.