# Arbitrary Bit-width Network: A Joint Layer-Wise Quantization and Adaptive Inference Approach

arXiv: https://arxiv.org/abs/2204.09992 · HTML: https://mlsanity.com/paper/2204.09992/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 6 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 3 of 3)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Fully Quantized Image Super-Resolution Networks](https://mlsanity.com/paper/2011.14265/) (2020)

## Built on by (most cited first, 11 of 11)

- [SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration](https://mlsanity.com/paper/2506.12723/) (2025)
- [Retraining-free Model Quantization via One-Shot Weight-Coupling Learning](https://mlsanity.com/paper/2401.01543/) (2024)
- [ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devices](https://mlsanity.com/paper/2303.09730/) (2023)
- [TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models](https://mlsanity.com/paper/2404.09532/) (2024)
- [PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference](https://mlsanity.com/paper/2407.05010/) (2024)
- [SEAM: Searching Transferable Mixed-Precision Quantization Policy through Large Margin Regularization](https://mlsanity.com/paper/2302.06845/) (2023)
- [MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization](https://mlsanity.com/paper/2305.08117/) (2023)
- [One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments](https://mlsanity.com/paper/2405.20202/) (2024)
- [DynaMIX: Resource Optimization for DNN-Based Real-Time Applications on a Multi-Tasking System](https://mlsanity.com/paper/2302.01568/) (2023)
- [RTF-Q: Efficient Unsupervised Domain Adaptation with Retraining-free Quantization](https://mlsanity.com/paper/2408.05752/) (2024)
- [Nearly Lossless Adaptive Bit Switching](https://mlsanity.com/paper/2502.01199/) (2025)

## Abstract (arXiv)

Conventional model quantization methods use a fixed quantization scheme to different data samples, which ignores the inherent "recognition difficulty" differences between various samples. We propose to feed different data samples with varying quantization schemes to achieve a data-dependent dynamic inference, at a fine-grained layer level. However, enabling this adaptive inference with changeable layer-wise quantization schemes is challenging because the combination of bit-widths and layers is growing exponentially, making it extremely difficult to train a single model in such a vast searching space and use it in practice. To solve this problem, we present the Arbitrary Bit-width Network (ABN), where the bit-widths of a single deep network can change at runtime for different data samples, with a layer-wise granularity. Specifically, first we build a weight-shared layer-wise quantizable "super-network" in which each layer can be allocated with multiple bit-widths and thus quantized differently on demand. The super-network provides a considerably large number of combinations of bit-widths and layers, each of which can be used during inference without retraining or storing myriad models. Second, based on the well-trained super-network, each layer's runtime bit-width selection decision is modeled as a Markov Decision Process (MDP) and solved by an adaptive inference strategy accordingly. Experiments show that the super-network can be built without accuracy degradation, and the bit-widths allocation of each layer can be adjusted to deal with various inputs on the fly. On ImageNet classification, we achieve 1.1% top1 accuracy improvement while saving 36.2% BitOps.