# Moonshine: Distilling with Cheap Convolutions

arXiv: https://arxiv.org/abs/1711.02613 · HTML: https://mlsanity.com/paper/1711.02613/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2017.

## Built on by (most cited first, 25 of 25)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Real-Time Video Generation with Pyramid Attention Broadcast](https://mlsanity.com/paper/2408.12588/) (2024)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2109.05263/) (2021)
- [Source Inference Attacks: Beyond Membership Inference Attacks in Federated Learning](https://mlsanity.com/paper/2310.00222/) (2023)
- [Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2106.13686/) (2021)
- [Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment](https://mlsanity.com/paper/2208.05969/) (2022)
- [QD-BEV : Quantization-aware View-guided Distillation for Multi-view 3D Object Detection](https://mlsanity.com/paper/2308.10515/) (2023)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [BoolNet: Minimizing The Energy Consumption of Binary Neural Networks](https://mlsanity.com/paper/2106.06991/) (2021)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Safety and Performance, Why Not Both? Bi-Objective Optimized Model Compression against Heterogeneous Attacks Toward AI Software Deployment](https://mlsanity.com/paper/2401.00996/) (2024)
- [Im2win: An Efficient Convolution Paradigm on GPU](https://mlsanity.com/paper/2306.14316/) (2023)
- [AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting](https://mlsanity.com/paper/2405.08019/) (2024)
- [Im2win: Memory Efficient Convolution On SIMD Architectures](https://mlsanity.com/paper/2306.14320/) (2023)
- [A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation](https://mlsanity.com/paper/2202.10461/) (2022)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [AsymmNet: Towards ultralight convolution neural networks using asymmetrical bottlenecks](https://mlsanity.com/paper/2104.07770/) (2021)
- [An Interpretable Neuron Embedding for Static Knowledge Distillation](https://mlsanity.com/paper/2211.07647/) (2022)
- [DLAS: An Exploration and Assessment of the Deep Learning Acceleration Stack](https://mlsanity.com/paper/2311.08909/) (2023)
- [P2Seg: Pointly-supervised Segmentation via Mutual Distillation](https://mlsanity.com/paper/2401.09709/) (2024)

## Abstract (arXiv)

Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a student architecture that is a simple transformation of the teacher architecture: no redesign is needed, and the same hyperparameters can be used. Using attention transfer, we provide Pareto curves/tables for distillation of residual networks with four benchmark datasets, indicating the memory versus accuracy payoff. We show that substantial memory savings are possible with very little loss of accuracy, and confirm that distillation provides student network performance that is better than training that student architecture directly on data.