Can Neural Network Memorization Be Localized?

Recent efforts at explaining the interplay of memorization and generalization in deep overparametrized networks have posited that neural networks textitmemorize "hard" examples in the final few layers of the model. Memorization refers to the ability to correctly predict on textitatypical examples of the training set. In this work, we show that rather than being confined to individual layers, memorization is a phenomenon confined to a small set of neurons in various layers of the model. First, via three experimental sources of converging evidence, we find that most layers are redundant for the memorization of examples and the layers that contribute to example memorization are, in general, not the final layers. The three sources are textitgradient accounting (measuring the contribution to the gradient norms from memorized and clean examples), textitlayer rewinding (replacing specific model weights of a converged model with previous training checkpoints), and textitretraining (training rewound layers only on clean examples). Second, we ask a more generic question: can memorization be localized textitanywhere in a model? We discover that memorization is often confined to a small number of neurons or channels (around 5) of the model. Based on these insights we propose a new form of dropout -- textitexample-tied dropout that enables us to direct the memorization of examples to an apriori determined set of neurons. By dropping out these neurons, we are able to reduce the accuracy on memorized examples from 100%→3%, while also reducing the generalization gap.