# BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer

arXiv: https://arxiv.org/abs/2105.08952 · HTML: https://mlsanity.com/paper/2105.08952/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 30 of 38)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](https://mlsanity.com/paper/2004.09576/) (2020)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.00090/) (2018)
- [APQ: Joint Search for Network Architecture, Pruning and Quantization Policy](https://mlsanity.com/paper/2006.08509/) (2020)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [AdaBits: Neural Network Quantization with Adaptive Bit-Widths](https://mlsanity.com/paper/1912.09666/) (2019)
- [Robust Quantization: One Model to Rule Them All](https://mlsanity.com/paper/2002.07686/) (2020)
- [Neural Architecture Transfer](https://mlsanity.com/paper/2005.05859/) (2020)
- [QKD: Quantization-aware Knowledge Distillation](https://mlsanity.com/paper/1911.12491/) (2019)

## Built on by (most cited first, 20 of 20)

- [Make RepVGG Greater Again: A Quantization-aware Approach](https://mlsanity.com/paper/2212.01593/) (2022)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [EQ-Net: Elastic Quantization Neural Networks](https://mlsanity.com/paper/2308.07650/) (2023)
- [Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach](https://mlsanity.com/paper/2207.10188/) (2022)
- [QGen: On the Ability to Generalize in Quantization Aware Training](https://mlsanity.com/paper/2404.11769/) (2024)
- [FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search](https://mlsanity.com/paper/2308.03290/) (2023)
- [BOMP-NAS: Bayesian Optimization Mixed Precision NAS](https://mlsanity.com/paper/2301.11810/) (2023)
- [DeepBurning-MixQ: An Open Source Mixed-Precision Neural Network Accelerator Design Framework for FPGAs](https://mlsanity.com/paper/2308.11334/) (2023)
- [SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search](https://mlsanity.com/paper/2312.13301/) (2023)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [Post-Training Quantization for Energy Efficient Realization of Deep Neural Networks](https://mlsanity.com/paper/2210.07906/) (2022)
- [SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference](https://mlsanity.com/paper/2303.08308/) (2023)
- [Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adapter](https://mlsanity.com/paper/2403.20080/) (2024)
- [Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices](https://mlsanity.com/paper/2401.08965/) (2024)
- [Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge](https://mlsanity.com/paper/2401.12350/) (2024)
- [Towards Accurate Post-training Quantization for Reparameterized Models](https://mlsanity.com/paper/2402.16121/) (2024)
- [NAS-BNN: Neural Architecture Search for Binary Neural Networks](https://mlsanity.com/paper/2408.15484/) (2024)
- [Device-Conditioned Neural Architecture Search for Efficient Robotic Manipulation](https://mlsanity.com/paper/2604.10170/) (2026)
- [Toward Multi-Domain and Long-Tailed Quantization via Feature Alignment and Scaling](https://mlsanity.com/paper/2606.04920/) (2026)
- [RAMP: Robust Adaptive Mixed-Precision Quantization for Edge CPU Vision Models](https://mlsanity.com/paper/2609.28262/) (2026)

## Abstract (arXiv)

As the applications of deep learning models on edge devices increase at an accelerating pace, fast adaptation to various scenarios with varying resource constraints has become a crucial aspect of model deployment. As a result, model optimization strategies with adaptive configuration are becoming increasingly popular. While single-shot quantized neural architecture search enjoys flexibility in both model architecture and quantization policy, the combined search space comes with many challenges, including instability when training the weight-sharing supernet and difficulty in navigating the exponentially growing search space. Existing methods tend to either limit the architecture search space to a small set of options or limit the quantization policy search space to fixed precision policies. To this end, we propose BatchQuant, a robust quantizer formulation that allows fast and stable training of a compact, single-shot, mixed-precision, weight-sharing supernet. We employ BatchQuant to train a compact supernet (offering over 1076 quantized subnets) within substantially fewer GPU hours than previous methods. Our approach, Quantized-for-all (QFA), is the first to seamlessly extend one-shot weight-sharing NAS supernet to support subnets with arbitrary ultra-low bitwidth mixed-precision quantization policies without retraining. QFA opens up new possibilities in joint hardware-aware neural architecture search and quantization. We demonstrate the effectiveness of our method on ImageNet and achieve SOTA Top-1 accuracy under a low complexity constraint (<20 MFLOPs). The code and models will be made publicly available at https://github.com/bhpfelix/QFA.