mlsanityData to 14 Oct 2026

Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences

arXiv 2310.04621 ↗First posted 6 Oct 2023cs.HC cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

On-device machine learning (ML) promises to improve the privacy, responsiveness, and proliferation of new, intelligent user experiences by moving ML computation onto everyday personal devices. However, today's large ML models must be drastically compressed to run efficiently on-device, a hurtle that requires deep, yet currently niche expertise. To engage the broader human-centered ML community in on-device ML experiences, we present the results from an interview study with 30 experts at Apple that specialize in producing efficient models. We compile tacit knowledge that experts have developed through practical experience with model compression across different hardware platforms. Our findings offer pragmatic considerations missing from prior work, covering the design process, trade-offs, and technical strategies that go into creating efficient models. Finally, we distill design recommendations for tooling to help ease the difficulty of this work and bring on-device ML into to more widespread practice.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 71 of its 153 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

71 of 153 references.

References · 71

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2023A Survey of Large Language Modelscited by 2,576
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
38 more
  1. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  2. 2019The State of Sparsity in Deep Neural Networkscited by 476
  3. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  4. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  5. 2018Model compression via distillation and quantizationcited by 289
  6. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  7. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  8. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  9. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  10. 2019Explainable Machine Learning in Deploymentcited by 211
  11. 2019The What-If Tool: Interactive Probing of Machine Learning Modelscited by 207
  12. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  13. 2019What Do Compressed Deep Neural Networks Forget?cited by 171
  14. 2020Characterising Bias in Compressed Modelscited by 167
  15. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  16. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
  17. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  18. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  19. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  20. 2019FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learningcited by 93
  21. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 92
  22. 2019Machine Learning at the Network Edge: A Surveycited by 84
  23. 2019Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AIcited by 79
  24. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  25. 2020Human Factors in Model Interpretability: Industry Practices, Challenges, and Needscited by 74
  26. 2020How do Data Science Workers Collaborate? Roles, Workflows, and Toolscited by 64
  27. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  28. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 63
  29. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  30. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 62
  31. 2020CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualizationcited by 57
  32. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  33. 2018GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentationcited by 41
  34. 2020Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practicescited by 39
  35. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  36. 2018Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutionscited by 33
  37. 2021AutoDS: Towards Human-Centered Automation of Data Sciencecited by 32
  38. 2021Dynamic Resolution Networkcited by 31

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 5.

21 papers.

Sort:

By area:

  1. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 46
  2. 2024OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answeringcited by 10
  3. 2024Wordflow: Social Prompt Engineering for Large Language Modelscited by 9
  4. 2024Talaria: Interactively Optimizing Machine Learning Models for Efficient Inferencecited by 6
  5. 2024Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experimentscited by 4
  6. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  7. 2025Choose Your Model Size: Any Compression of Large Language Models Without Re-Computationcited by 3
  8. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  9. 2025On the Interaction of Compressibility and Adversarial Robustnesscited by 2
  10. 2026HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applicationscited by 2
  11. 2024Constraint Guided Model Quantization of Neural Networkscited by 1
  12. 2025Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Modelscited by 1
9 more
  1. 2025On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcited by 1
  2. 2024Mobile Fitting Room: On-device Virtual Try-on via Diffusion Modelscited by 0
  3. 2024Combining Relevance and Magnitude for Resource-Aware DNN Pruningcited by 0
  4. 2025Optimization of Layer Skipping and Frequency Scaling for Convolutional Neural Networks under Latency Constraintcited by 0
  5. 2025Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Codingcited by 0
  6. 2026ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Realitycited by 0
  7. 2026Weight Group-wise Post-Training Quantization for Medical Foundation Modelcited by 0
  8. 2026A Resource-Efficient Hybrid CNN-LSTM network for image-based bean leaf disease classificationcited by 0
  9. 2026Input Resolution Matters: Real-Time Object Detection Latencycited by 0