# Bit-Mixer: Mixed-precision networks with runtime bit-width selection

arXiv: https://arxiv.org/abs/2103.17267 · HTML: https://mlsanity.com/paper/2103.17267/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 50 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Modulating early visual processing by language](https://mlsanity.com/paper/1707.00683/) (2017)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.00090/) (2018)

## Built on by (most cited first, 22 of 22)

- [EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers](https://mlsanity.com/paper/2205.03436/) (2022)
- [SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications](https://mlsanity.com/paper/2303.15446/) (2023)
- [Progressive Mixed-Precision Decoding for Efficient LLM Inference](https://mlsanity.com/paper/2410.13461/) (2024)
- [Retraining-free Model Quantization via One-Shot Weight-Coupling Learning](https://mlsanity.com/paper/2401.01543/) (2024)
- [Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach](https://mlsanity.com/paper/2207.10188/) (2022)
- [Tiny Machine Learning for Concept Drift](https://mlsanity.com/paper/2107.14759/) (2021)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [BARVINN: Arbitrary Precision DNN Accelerator Controlled by a RISC-V CPU](https://mlsanity.com/paper/2301.00290/) (2022)
- [MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization](https://mlsanity.com/paper/2305.08117/) (2023)
- [DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment](https://mlsanity.com/paper/2508.06041/) (2025)
- [BF-IMNA: A Bit Fluid In-Memory Neural Architecture for Neural Network Acceleration](https://mlsanity.com/paper/2411.01417/) (2024)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [FraQAT: Quantization Aware Training with Fractional bits](https://mlsanity.com/paper/2510.14823/) (2025)
- [MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2607.23047/) (2026)
- [A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank](https://mlsanity.com/paper/2111.15179/) (2021)
- [A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness](https://mlsanity.com/paper/2204.00426/) (2022)
- [Learning Accurate Performance Predictors for Ultrafast Automated Model Compression](https://mlsanity.com/paper/2304.06393/) (2023)
- [Constraint Guided Model Quantization of Neural Networks](https://mlsanity.com/paper/2409.20138/) (2024)
- [SigmaQuant: Hardware-Aware Heterogeneous Quantization Method for Edge DNN Inference](https://mlsanity.com/paper/2602.22136/) (2026)
- [MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference](https://mlsanity.com/paper/2604.00529/) (2026)
- [DynaMIX: Resource Optimization for DNN-Based Real-Time Applications on a Multi-Tasking System](https://mlsanity.com/paper/2302.01568/) (2023)
- [Nearly Lossless Adaptive Bit Switching](https://mlsanity.com/paper/2502.01199/) (2025)

## Abstract (arXiv)

Mixed-precision networks allow for a variable bit-width quantization for every layer in the network. A major limitation of existing work is that the bit-width for each layer must be predefined during training time. This allows little flexibility if the characteristics of the device on which the network is deployed change during runtime. In this work, we propose Bit-Mixer, the very first method to train a meta-quantized network where during test time any layer can change its bid-width without affecting at all the overall network's ability for highly accurate inference. To this end, we make 2 key contributions: (a) Transitional Batch-Norms, and (b) a 3-stage optimization process which is shown capable of training such a network. We show that our method can result in mixed precision networks that exhibit the desirable flexibility properties for on-device deployment without compromising accuracy. Code will be made available.