mlsanityData to 7 Oct 2026

Sharp asymptotics on the compression of two-layer neural networks

arXiv 2205.08199 ↗First posted 17 May 2022cs.IT cs.LG math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M<N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimize the population L_2 loss between the outputs of the target and of the compressed network, under the assumption of Gaussian inputs. By using tools from high-dimensional probability, we show that this non-convex problem can be simplified when the target network is sufficiently over-parameterized, and provide the error rate of this approximation as a function of the input dimension and N. In this mean-field limit, the simplified objective, as well as the optimal weights of the compressed network, does not depend on the realization of the target network, but only on expected scaling factors. Furthermore, for networks with ReLU activation, we conjecture that the optimum of the simplified optimization problem is achieved by taking weights on the Equiangular Tight Frame (ETF), while the scaling of the weights and the orientation of the ETF depend on the parameters of the target network. Numerical evidence is provided to support this conjecture.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 17 May 2022.

Builds on

18 of 25 references.

References · 18

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  3. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  4. 2019The State of Sparsity in Deep Neural Networkscited by 471
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  8. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  9. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
  10. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  11. 2020Good Subnetworks Provably Exist: Pruning via Greedy Forward Selectioncited by 70
  12. 2016Compression of Neural Machine Translation Models via Pruningcited by 43
6 more
  1. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  2. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
  3. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  4. 2020Neuron Merging: Compensating for Pruned Neuronscited by 24
  5. 2018On the Connection Between Learning Two-Layers Neural Networks and Tensor Decompositioncited by 14
  6. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13

Built on by

Not cited yet.