Non-identifiability distinguishes Neural Networks among Parametric Models

One of the enduring problems surrounding neural networks is to identify the factors that differentiate them from traditional statistical models. We prove a pair of results which distinguish feedforward neural networks among parametric models at the population level, for regression tasks. Firstly, we prove that for any pair of random variables (X,Y), neural networks always learn a nontrivial relationship between X and Y, if one exists. Secondly, we prove that for reasonable smooth parametric models, under local and global identifiability conditions, there exists a nontrivial (X,Y) pair for which the parametric model learns the constant predictor mathbbE[Y]. Together, our results suggest that a lack of identifiability distinguishes neural networks among the class of smooth parametric models.