The Efficacy of L1 Regularization in Two-Layer Neural Networks

A crucial problem in neural networks is to select the most appropriate number of hidden neurons and obtain tight statistical risk bounds. In this work, we present a new perspective towards the bias-variance tradeoff in neural networks. As an alternative to selecting the number of neurons, we theoretically show that L1 regularization can control the generalization error and sparsify the input dimension. In particular, with an appropriate L1 regularization on the output layer, the network can produce a statistical risk that is near minimax optimal. Moreover, an appropriate L1 regularization on the input layer leads to a risk bound that does not involve the input data dimension. Our analysis is based on a new amalgamation of dimension-based and norm-based complexity analysis to bound the generalization error. A consequent observation from our results is that an excessively large number of neurons do not necessarily inflate generalization errors under a suitable regularization.