mlsanityData to 14 Oct 2026

Width-Independent Compressibility of Deep Neural Networks

arXiv 2608.21752 ↗First posted 22 Aug 2026cs.LG cond-mat.dis-nn cs.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

It has long been known that well-trained neural networks can be compressed very strongly without affecting their performance, an important phenomenon that remains poorly understood. We prove a uniform compressibility theorem for deep multilayer perceptrons with analytic activations. For a deep, wide fixed teacher network, there exists a narrow (same depth) network that approximately represents the same function as the original. The reachable compressed width is strikingly independent of the original width, but is O((log(1/varepsilon))^din), where varepsilon is the error budget and din is the effective input dimension. Our construction involves a novel derivative-matching technique which is aware of the low-dimensional input, and a layer-wise reweighting that preserves the input-output mapping.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

31 of 56 references.

References · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2020What is the State of Neural Network Pruning?cited by 583
  7. 2016Error bounds for approximations with deep ReLU networkscited by 574
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  10. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
19 more
  1. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  2. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  3. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  4. 2016Benefits of depth in neural networkscited by 250
  5. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  6. 2019Universal Approximation with Deep Narrow Networkscited by 200
  7. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  8. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
  9. 2020Deep Network Approximation for Smooth Functionscited by 164
  10. 2021Towards Understanding Knowledge Distillationcited by 158
  11. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  12. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  13. 2018Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Boundscited by 43
  14. 2022Width is Less Important than Depth in ReLU Neural Networkscited by 10
  15. 2022A Theoretical Understanding of Neural Network Compression from Sparse Linear Approximationcited by 5
  16. 2023Pruning Before Training May Improve Generalization, Provablycited by 3
  17. 2025A universal compression theory for lottery ticket hypothesis and neural scaling lawscited by 2
  18. 2026Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activationscited by 2
  19. 2025Theoretical Compression Bounds for Wide Multilayer Perceptronscited by 1

Built on by

Not cited yet.