Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation

A common technique for compressing a neural network is to compute the k-rank ℓ2 approximation Ak,2 of the matrix A∈ℝn× d that corresponds to a fully connected layer (or embedding layer). Here, d is the number of the neurons in the layer, n is the number in the next one, and Ak,2 can be stored in O((n+d)k) memory instead of O(nd). This ℓ2-approximation minimizes the sum over every entry to the power of p=2 in the matrix A - Ak,2, among every matrix Ak,2∈ℝn× d whose rank is k. While it can be computed efficiently via SVD, the ℓ2-approximation is known to be very sensitive to outliers ("far-away" rows). Hence, machine learning uses e.g. Lasso Regression, ℓ1-regularization, and ℓ1-SVM that use the ℓ1-norm. This paper suggests to replace the k-rank ℓ2 approximation by ℓp, for p∈ [1,2]. We then provide practical and provable approximation algorithms to compute it for any p≥1, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves 28% compression of RoBERTa's embedding layer with only 0.63% additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to 11% drop using the existing ℓ2-approximation. Open code is provided for reproducing and extending our results.