# LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time

arXiv: https://arxiv.org/abs/2110.04252 · HTML: https://mlsanity.com/paper/2110.04252/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 26 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Learning Neural Network Subspaces](https://mlsanity.com/paper/2102.10472/) (2021)
- [Exploring Sparsity in Recurrent Neural Networks](https://mlsanity.com/paper/1704.05119/) (2017)
- [On-Device Machine Learning: An Algorithms and Learning Theory Perspective](https://mlsanity.com/paper/1911.00623/) (2019)
- [Compression of Neural Machine Translation Models via Pruning](https://mlsanity.com/paper/1606.09274/) (2016)
- [Layer-Wise Data-Free CNN Compression](https://mlsanity.com/paper/2011.09058/) (2020)
- [Switchable Precision Neural Networks](https://mlsanity.com/paper/2002.02815/) (2020)

## Built on by (most cited first, 3 of 3)

- [All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Management](https://mlsanity.com/paper/2212.05122/) (2022)
- [Low-Loss Subspace Compression for Clean Gains against Multi-Agent Backdoor Attacks](https://mlsanity.com/paper/2203.03692/) (2022)
- [Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems](https://mlsanity.com/paper/2306.12691/) (2023)

## Abstract (arXiv)

When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees. Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a "compressible subspace" of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time. We present results for achieving arbitrarily fine-grained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high accuracy for sparsity rates above 90% when testing our compressed models. We also demonstrate that our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.