# Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences

arXiv: https://arxiv.org/abs/2310.04621 · HTML: https://mlsanity.com/paper/2310.04621/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 153 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 30 of 71)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Characterising Bias in Compressed Models](https://mlsanity.com/paper/2010.03058/) (2020)
- [FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization](https://mlsanity.com/paper/2303.14189/) (2023)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Soft Threshold Weight Reparameterization for Learnable Sparsity](https://mlsanity.com/paper/2002.03231/) (2020)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)

## Built on by (most cited first, 21 of 21)

- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering](https://mlsanity.com/paper/2409.08250/) (2024)
- [Wordflow: Social Prompt Engineering for Large Language Models](https://mlsanity.com/paper/2401.14447/) (2024)
- [Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference](https://mlsanity.com/paper/2404.03085/) (2024)
- [Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experiments](https://mlsanity.com/paper/2408.03274/) (2024)
- [PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing](https://mlsanity.com/paper/2503.12167/) (2025)
- [Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation](https://mlsanity.com/paper/2502.01717/) (2025)
- [From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap](https://mlsanity.com/paper/2410.20791/) (2024)
- [On the Interaction of Compressibility and Adversarial Robustness](https://mlsanity.com/paper/2507.17725/) (2025)
- [HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications](https://mlsanity.com/paper/2601.12610/) (2026)
- [Constraint Guided Model Quantization of Neural Networks](https://mlsanity.com/paper/2409.20138/) (2024)
- [Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models](https://mlsanity.com/paper/2502.00046/) (2025)
- [On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization](https://mlsanity.com/paper/2507.09428/) (2025)
- [Mobile Fitting Room: On-device Virtual Try-on via Diffusion Models](https://mlsanity.com/paper/2402.01877/) (2024)
- [Combining Relevance and Magnitude for Resource-Aware DNN Pruning](https://mlsanity.com/paper/2405.13088/) (2024)
- [Optimization of Layer Skipping and Frequency Scaling for Convolutional Neural Networks under Latency Constraint](https://mlsanity.com/paper/2503.24014/) (2025)
- [Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding](https://mlsanity.com/paper/2505.18758/) (2025)
- [ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality](https://mlsanity.com/paper/2604.04905/) (2026)
- [Weight Group-wise Post-Training Quantization for Medical Foundation Model](https://mlsanity.com/paper/2604.07674/) (2026)
- [A Resource-Efficient Hybrid CNN-LSTM network for image-based bean leaf disease classification](https://mlsanity.com/paper/2604.13835/) (2026)
- [Input Resolution Matters: Real-Time Object Detection Latency](https://mlsanity.com/paper/2609.12920/) (2026)

## Abstract (arXiv)

On-device machine learning (ML) promises to improve the privacy, responsiveness, and proliferation of new, intelligent user experiences by moving ML computation onto everyday personal devices. However, today's large ML models must be drastically compressed to run efficiently on-device, a hurtle that requires deep, yet currently niche expertise. To engage the broader human-centered ML community in on-device ML experiences, we present the results from an interview study with 30 experts at Apple that specialize in producing efficient models. We compile tacit knowledge that experts have developed through practical experience with model compression across different hardware platforms. Our findings offer pragmatic considerations missing from prior work, covering the design process, trade-offs, and technical strategies that go into creating efficient models. Finally, we distill design recommendations for tooling to help ease the difficulty of this work and bring on-device ML into to more widespread practice.