# SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search

arXiv: https://arxiv.org/abs/2312.13301 · HTML: https://mlsanity.com/paper/2312.13301/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization](https://mlsanity.com/paper/2111.12293/) (2021)
- [Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.00090/) (2018)
- [APQ: Joint Search for Network Architecture, Pruning and Quantization Policy](https://mlsanity.com/paper/2006.08509/) (2020)
- [Rethinking Differentiable Search for Mixed-Precision Neural Networks](https://mlsanity.com/paper/2004.05795/) (2020)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer](https://mlsanity.com/paper/2105.08952/) (2021)
- [Automatic Mixed-Precision Quantization Search of BERT](https://mlsanity.com/paper/2112.14938/) (2021)
- [ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks](https://mlsanity.com/paper/1811.01704/) (2018)
- [Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach](https://mlsanity.com/paper/2207.10188/) (2022)
- [A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities](https://mlsanity.com/paper/2205.10358/) (2022)
- [FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search](https://mlsanity.com/paper/2308.03290/) (2023)
- [InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning](https://mlsanity.com/paper/2308.15609/) (2023)

## Built on by (most cited first, 2 of 2)

- [Arithmetic-Intensity-Aware Quantization](https://mlsanity.com/paper/2512.14090/) (2025)
- [NAS-Driven Hardware Accelerator Exploration for Edge AI and Quantization Effects on the Pareto Space](https://mlsanity.com/paper/2608.13293/) (2026)

## Abstract (arXiv)

Recent one-shot Neural Architecture Search algorithms rely on training a hardware-agnostic super-network tailored to a specific task and then extracting efficient sub-networks for different hardware platforms. Popular approaches separate the training of super-networks from the search for sub-networks, often employing predictors to alleviate the computational overhead associated with search. Additionally, certain methods also incorporate the quantization policy within the search space. However, while the quantization policy search for convolutional neural networks is well studied, the extension of these methods to transformers and especially foundation models remains under-explored. In this paper, we demonstrate that by using multi-objective search algorithms paired with lightly trained predictors, we can efficiently search for both the sub-network architecture and the corresponding quantization policy and outperform their respective baselines across different performance objectives such as accuracy, model size, and latency. Specifically, we demonstrate that our approach performs well across both uni-modal (ViT and BERT) and multi-modal (BEiT-3) transformer-based architectures as well as convolutional architectures (ResNet). For certain networks, we demonstrate an improvement of up to 4.80x and 3.44x for latency and model size respectively, without degradation in accuracy compared to the fully quantized INT8 baselines.