mlsanityData to 14 Oct 2026

On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance

arXiv 2411.00907 ↗First posted 1 Nov 2024cs.DC cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

To help MLOps engineers decide which operator to use in which deployment scenario, this study aims to empirically assess the accuracy vs latency trade-off of white-box (training-based) and black-box operators (non-training-based) and their combinations in an Edge AI setup. We perform inference experiments including 3 white-box (i.e., QAT, Pruning, Knowledge Distillation), 2 black-box (i.e., Partition, SPTQ), and their combined operators (i.e., Distilled SPTQ, SPTQ Partition) across 3 tiers (i.e., Mobile, Edge, Cloud) on 4 commonly-used Computer Vision and Natural Language Processing models to identify the effective strategies, considering the perspective of MLOps Engineers. Our Results indicate that the combination of Distillation and SPTQ operators (i.e., DSPTQ) should be preferred over non-hybrid operators when lower latency is required in the edge at small to medium accuracy drop. Among the non-hybrid operators, the Distilled operator is a better alternative in both mobile and edge tiers for lower latency performance at the cost of small to medium accuracy loss. Moreover, the operators involving distillation show lower latency in resource-constrained tiers (Mobile, Edge) compared to the operators involving Partitioning across Mobile and Edge tiers. For textual subject models, which have low input data size requirements, the Cloud tier is a better alternative for the deployment of operators than the Mobile, Edge, or Mobile-Edge tier (the latter being used for operators involving partitioning). In contrast, for image-based subject models, which have high input data size requirements, the Edge tier is a better alternative for operators than Mobile, Edge, or their combination.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 73 of its 133 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

73 of 133 references.

References · 73

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016Wide Residual Networkscited by 3,070
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
38 more
  1. 2019Learned Step Size Quantizationcited by 628
  2. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  3. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  4. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  5. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  6. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  7. 2015Deep Learning with Limited Numerical Precisioncited by 376
  8. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  9. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  10. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  11. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  12. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  13. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  14. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  15. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  16. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 143
  17. 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 134
  18. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  19. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  20. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 96
  21. 2020Bayesian Bits: Unifying Quantization and Pruningcited by 94
  22. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 92
  23. 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 90
  24. 2019Machine Learning at the Network Edge: A Surveycited by 84
  25. 2020Understanding and Improving Knowledge Distillationcited by 83
  26. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 79
  27. 2019Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligencecited by 76
  28. 2018NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Visioncited by 68
  29. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  30. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  31. 2016ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGAcited by 52
  32. 2018Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergycited by 51
  33. 2023Pruning vs Quantization: Which is Better?cited by 46
  34. 2019Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorizationcited by 45
  35. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  36. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  37. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  38. 2021Supervised Compression for Resource-Constrained Edge Computing Systemscited by 31

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Achieving Trustworthy Real-Time Decision Support Systems with Low-Latency Interpretable AI Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026EdgeFM: Efficient Edge Inference for Vision-Language Models
  2. 2026A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
  3. 2026Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMs
  4. 2025Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
  5. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deployment
  6. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environments
  7. 2021ML-EXray: Visibility into ML Deployment on the Edge
  8. 2021RCT: Resource Constrained Training for Edge AI