mlsanityData to 13 Oct 2026

Bit-Mixer: Mixed-precision networks with runtime bit-width selection

arXiv 2103.17267 ↗First posted 31 Mar 2021cs.LG cs.AI cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Mixed-precision networks allow for a variable bit-width quantization for every layer in the network. A major limitation of existing work is that the bit-width for each layer must be predefined during training time. This allows little flexibility if the characteristics of the device on which the network is deployed change during runtime. In this work, we propose Bit-Mixer, the very first method to train a meta-quantized network where during test time any layer can change its bid-width without affecting at all the overall network's ability for highly accurate inference. To this end, we make 2 key contributions: (a) Transitional Batch-Norms, and (b) a 3-stage optimization process which is shown capable of training such a network. We show that our method can result in mixed precision networks that exhibit the desirable flexibility properties for on-device deployment without compromising accuracy. Code will be made available.

Key facts. Cited by 21 papers in the map (data through 13 Oct 2026). Builds on 45 of its 50 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

45 of 50 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,315
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,789
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,585
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,138
  10. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  11. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 783
  12. 2019Learned Step Size Quantizationcited by 621
33 more
  1. 2018Rethinking the Value of Network Pruningcited by 597
  2. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 589
  3. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 565
  4. 2019Importance Estimation for Neural Network Pruningcited by 536
  5. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 465
  6. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 439
  7. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 399
  8. 2018Slimmable Neural Networkscited by 302
  9. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  10. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  11. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  12. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  13. 2016Trained Ternary Quantizationcited by 208
  14. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 201
  15. 2017Modulating early visual processing by languagecited by 167
  16. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  17. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  18. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 125
  19. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  20. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  21. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  22. 2019XNOR-Net++: Improved Binary Neural Networkscited by 97
  23. 2019Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hourscited by 87
  24. 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 82
  25. 2019AdaBits: Neural Network Quantization with Adaptive Bit-Widthscited by 63
  26. 2019Learning Filter Basis for Convolutional Neural Network Compressioncited by 57
  27. 2019Training Quantized Neural Networks with a Full-precision Auxiliary Modulecited by 35
  28. 2020High-Capacity Expert Binary Networkscited by 34
  29. 2020Resolution Switchable Networks for Runtime Efficient Image Recognitioncited by 19
  30. 2019Mixed Precision DNNs: All you need is a good parametrizationcited by 18
  31. 2017Ternary Neural Networks with Fine-Grained Quantizationcited by 17
  32. 2019Towards Efficient Training for Neural Network Quantizationcited by 16
  33. 2018Hierarchical binary CNNs for landmark localization with limited resourcescited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 4; 2024: 4; 2025: 3; 2026: 2.

21 papers.

Sort:

By area:

  1. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  2. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  3. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inferencecited by 16
  4. 2024Retraining-free Model Quantization via One-Shot Weight-Coupling Learningcited by 13
  5. 2022Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approachcited by 9
  6. 2021Tiny Machine Learning for Concept Driftcited by 7
  7. 2022Mixed-Precision Neural Networks: A Surveycited by 7
  8. 2022BARVINN: Arbitrary Precision DNN Accelerator Controlled by a RISC-V CPUcited by 6
  9. 2023MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantizationcited by 6
  10. 2025DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignmentcited by 4
  11. 2024BF-IMNA: A Bit Fluid In-Memory Neural Architecture for Neural Network Accelerationcited by 3
  12. 2022Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Reviewcited by 2
9 more
  1. 2025FraQAT: Quantization Aware Training with Fractional bitscited by 2
  2. 2026MixQuant: Adaptive Mixed-Precision Quantization for Large Language Modelscited by 2
  3. 2021A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rankcited by 1
  4. 2022A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustnesscited by 1
  5. 2023Learning Accurate Performance Predictors for Ultrafast Automated Model Compressioncited by 1
  6. 2024Constraint Guided Model Quantization of Neural Networkscited by 1
  7. 2026SigmaQuant: Hardware-Aware Heterogeneous Quantization Method for Edge DNN Inferencecited by 1
  8. 2023DynaMIX: Resource Optimization for DNN-Based Real-Time Applications on a Multi-Tasking Systemcited by 0
  9. 2025Nearly Lossless Adaptive Bit Switchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Practical Mixed Precision Algorithm for Post-Training Quantization
  2. 2023Free Bits: Latency Optimization of Mixed-Precision Quantized Neural Networks on the Edge
  3. 2022AMED: Automatic Mixed-Precision Quantization for Edge Devices
  4. 2022Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes
  5. 2025InfoQ: Mixed-Precision Quantization via Global Information Flow
  6. 2023MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization
  7. 2023MixQuant: Mixed Precision Quantization with a Bit-width Optimization Search
  8. 2026Task-Aware QUBO Allocation for Mixed-Precision Quantization