# QPART: Adaptive Model Quantization and Dynamic Workload Balancing for Accuracy-aware Edge Inference

arXiv: https://arxiv.org/abs/2506.23934 · HTML: https://mlsanity.com/paper/2506.23934/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 60 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Federated Learning over Wireless Fading Channels](https://mlsanity.com/paper/1907.09769/) (2019)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Adaptive Quantization for Deep Neural Network](https://mlsanity.com/paper/1712.01048/) (2017)
- [Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead](https://mlsanity.com/paper/2105.09121/) (2021)
- [Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning](https://mlsanity.com/paper/1903.03472/) (2019)
- [FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things](https://mlsanity.com/paper/1911.03314/) (2019)
- [1xN Pattern for Pruning Convolutional Neural Networks](https://mlsanity.com/paper/2105.14713/) (2021)
- [Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence](https://mlsanity.com/paper/1906.01493/) (2019)
- [Energy Consumption of Neural Networks on NVIDIA Edge Boards: an Empirical Model](https://mlsanity.com/paper/2210.01625/) (2022)
- [Exploring Edge TPU for Network Intrusion Detection in IoT](https://mlsanity.com/paper/2103.16295/) (2021)
- [Knowledge Distillation for Mobile Edge Computation Offloading](https://mlsanity.com/paper/2004.04366/) (2020)
- [Graph Reinforcement Learning-based CNN Inference Offloading in Dynamic Edge Computing](https://mlsanity.com/paper/2210.13464/) (2022)
- [Selective Task offloading for Maximum Inference Accuracy and Energy efficient Real-Time IoT Sensing Systems](https://mlsanity.com/paper/2402.16904/) (2024)

## Built on by (most cited first, 1 of 1)

- [Time-Series at the Edge: Tiny Separable CNNs for Wearable Gait Detection and Optimal Sensor Placement](https://mlsanity.com/paper/2512.00396/) (2025)

## Abstract (arXiv)

As machine learning inferences increasingly move to edge devices, adapting to diverse computational capabilities, hardware, and memory constraints becomes more critical. Instead of relying on a pre-trained model fixed for all future inference queries across diverse edge devices, we argue that planning an inference pattern with a request-specific model tailored to the device's computational capacity, accuracy requirements, and time constraints is more cost-efficient and robust to diverse scenarios. To this end, we propose an accuracy-aware and workload-balanced inference system that integrates joint model quantization and inference partitioning. In this approach, the server dynamically responds to inference queries by sending a quantized model and adaptively sharing the inference workload with the device. Meanwhile, the device's computational power, channel capacity, and accuracy requirements are considered when deciding. Furthermore, we introduce a new optimization framework for the inference system, incorporating joint model quantization and partitioning. Our approach optimizes layer-wise quantization bit width and partition points to minimize time consumption and cost while accounting for varying accuracy requirements of tasks through an accuracy degradation metric in our optimization model. To our knowledge, this work represents the first exploration of optimizing quantization layer-wise bit-width in the inference serving system, by introducing theoretical measurement of accuracy degradation. Simulation results demonstrate a substantial reduction in overall time and power consumption, with computation payloads decreasing by over 80% and accuracy degradation kept below 1%.