# Neural Network Compression for Noisy Storage Devices

arXiv: https://arxiv.org/abs/2102.07725 · HTML: https://mlsanity.com/paper/2102.07725/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 84 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Characterising Bias in Compressed Models](https://mlsanity.com/paper/2010.03058/) (2020)
- [Training with Quantization Noise for Extreme Model Compression](https://mlsanity.com/paper/2004.07320/) (2020)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [A Kronecker-factored approximate Fisher matrix for convolution layers](https://mlsanity.com/paper/1602.01407/) (2016)
- [Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach](https://mlsanity.com/paper/1804.05862/) (2018)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters](https://mlsanity.com/paper/1810.00440/) (2018)
- [Where is the Information in a Deep Neural Network?](https://mlsanity.com/paper/1905.12213/) (2019)
- [A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs](https://mlsanity.com/paper/2011.03083/) (2020)
- [Scalable Model Compression by Entropy Penalized Reparameterization](https://mlsanity.com/paper/1906.06624/) (2019)
- [rTop-k: A Statistical Estimation Approach to Distributed SGD](https://mlsanity.com/paper/2005.10761/) (2020)
- [DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks](https://mlsanity.com/paper/1907.11900/) (2019)
- [WoodFisher: Efficient Second-Order Approximation for Neural Network Compression](https://mlsanity.com/paper/2004.14340/) (2020)
- [Transform Quantization for CNN (Convolutional Neural Network) Compression](https://mlsanity.com/paper/2009.01174/) (2020)
- [Universal Deep Neural Network Compression](https://mlsanity.com/paper/1802.02271/) (2018)
- [Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation](https://mlsanity.com/paper/2001.04974/) (2020)

## Built on by (most cited first, 8 of 8)

- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)
- [LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks](https://mlsanity.com/paper/2111.08988/) (2021)
- [Neural 3D Scene Compression via Model Compression](https://mlsanity.com/paper/2105.03120/) (2021)
- [Denoising Noisy Neural Networks: A Bayesian Approach with Compensation](https://mlsanity.com/paper/2105.10699/) (2021)
- [Lossy and Lossless (L2) Post-training Model Size Compression](https://mlsanity.com/paper/2308.04269/) (2023)
- [SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruning](https://mlsanity.com/paper/2310.04918/) (2023)
- [Random resistive memory-based deep extreme point learning machine for unified visual processing](https://mlsanity.com/paper/2312.09262/) (2023)
- [Over-the-Air Multi-Sensor Inference with Neural Networks Using Memristor-Based Analog Computing](https://mlsanity.com/paper/2501.10245/) (2025)

## Abstract (arXiv)

Compression and efficient storage of neural network (NN) parameters is critical for applications that run on resource-constrained devices. Despite the significant progress in NN model compression, there has been considerably less investigation in the actual textitphysical storage of NN parameters. Conventionally, model compression and physical storage are decoupled, as digital storage media with error-correcting codes (ECCs) provide robust error-free storage. However, this decoupled approach is inefficient as it ignores the overparameterization present in most NNs and forces the memory device to allocate the same amount of resources to every bit of information regardless of its importance. In this work, we investigate analog memory devices as an alternative to digital media -- one that naturally provides a way to add more protection for significant bits unlike its counterpart, but is noisy and may compromise the stored model's performance if used naively. We develop a variety of robust coding strategies for NN weight storage on analog devices, and propose an approach to jointly optimize model compression and memory resource allocation. We then demonstrate the efficacy of our approach on models trained on MNIST, CIFAR-10 and ImageNet datasets for existing compression techniques. Compared to conventional error-free digital storage, our method reduces the memory footprint by up to one order of magnitude, without significantly compromising the stored model's accuracy.