The Local Elasticity of Neural Networks

This paper presents a phenomenon in neural networks that we refer to as textitlocal elasticity. Roughly speaking, a classifier is said to be locally elastic if its prediction at a feature vector bx' is textitnot significantly perturbed, after the classifier is updated via stochastic gradient descent at a (labeled) feature vector bx that is textitdissimilar to bx' in a certain sense. This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers. In addition, we offer a geometric interpretation of local elasticity using the neural tangent kernel citepjacot2018neural. Building on top of local elasticity, we obtain pairwise similarity measures between feature vectors, which can be used for clustering in conjunction with K-means. The effectiveness of the clustering algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the hypothesis of local elasticity of neural networks on real-life data. Finally, we discuss some implications of local elasticity to shed light on several intriguing aspects of deep neural networks.