mlsanityData to 7 Oct 2026

On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performance

arXiv 2403.17154 ↗First posted 25 Mar 2024cs.SE cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Deciding what combination of operators to use across the Edge AI tiers to achieve specific latency and model performance requirements is an open question for MLOps engineers. This study aims to empirically assess the accuracy vs inference time trade-off of different black-box Edge AI deployment strategies, i.e., combinations of deployment operators and deployment tiers. In this paper, we conduct inference experiments involving 3 deployment operators (i.e., Partitioning, Quantization, Early Exit), 3 deployment tiers (i.e., Mobile, Edge, Cloud) and their combinations on four widely used Computer-Vision models to investigate the optimal strategies from the point of view of MLOps developers. Our findings suggest that Edge deployment using the hybrid Quantization + Early Exit operator could be preferred over non-hybrid operators (Quantization/Early Exit on Edge, Partition on Mobile-Edge) when faster latency is a concern at medium accuracy loss. However, when minimizing accuracy loss is a concern, MLOps engineers should prefer using only a Quantization operator on edge at a latency reduction or increase, respectively over the Early Exit/Partition (on edge/mobile-edge) and Quantized Early Exit (on edge) operators. In scenarios constrained by Mobile CPU/RAM resources, a preference for Partitioning across mobile and edge tiers is observed over mobile deployment. For models with smaller input data samples (such as FCN), a network-constrained cloud deployment can also be a better alternative than Mobile/Edge deployment and Partitioning strategies. For models with large input data samples (ResNet, ResNext, DUC), an edge tier having higher network/computational capabilities than Cloud/Mobile can be a more viable option than Partitioning and Mobile/Cloud deployment strategies.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 63 of its 117 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

63 of 117 references.

References · 63

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  8. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  9. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  10. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  11. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  12. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
38 more
  1. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  2. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  3. 2019Depth-Adaptive Transformercited by 227
  4. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  5. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  6. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  7. 2017Understanding Convolution for Semantic Segmentationcited by 168
  8. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  9. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  10. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  11. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  12. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 119
  13. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  14. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109
  15. 2019BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systemscited by 94
  16. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 93
  17. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  18. 2019Machine Learning at the Network Edge: A Surveycited by 83
  19. 2018JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Servicescited by 79
  20. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  21. 2019Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligencecited by 75
  22. 2020Why should we add early exits to neural networks?cited by 70
  23. 2019BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Servicescited by 67
  24. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  25. 2018Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergycited by 50
  26. 2018Deep feature compression for collaborative object detectioncited by 48
  27. 2019Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorizationcited by 45
  28. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  29. 2019Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inferencecited by 31
  30. 2021Supervised Compression for Resource-Constrained Edge Computing Systemscited by 31
  31. 2020Early Exit or Not: Resource-Efficient Blind Quality Enhancement for Compressed Imagescited by 28
  32. 2020Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networkscited by 28
  33. 2022BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computingcited by 28
  34. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26
  35. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  36. 2020Bringing AI To Edge: From Deep Learning's Perspectivecited by 24
  37. 2018Quantization for Rapid Deployment of Deep Neural Networkscited by 20
  38. 2018Auto-tuning Neural Network Quantization Framework for Collaborative Inference Between the Cloud and Edgecited by 20

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2025Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Surveycited by 13
  2. 2024On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performancecited by 1
  3. 2024Insights into resource utilization of code small language models serving with runtime engines and execution providerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
  2. 2026Optimizing AI Inference Across the Deployment Stack
  3. 2026EdgeFM: Efficient Edge Inference for Vision-Language Models
  4. 2025Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
  5. 2022Resource-Constrained Edge AI with Early Exit Prediction
  6. 2026From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
  7. 2026HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference
  8. 2026EdgeVLN: Runtime-Aware Deployment Ready Quantized Vision Language Navigation Model