# Rethinking Differentiable Search for Mixed-Precision Neural Networks

arXiv: https://arxiv.org/abs/2004.05795 · HTML: https://mlsanity.com/paper/2004.05795/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 30 of 82)

- [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023)
- [Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling](https://mlsanity.com/paper/2304.09145/) (2023)
- [Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance](https://mlsanity.com/paper/2203.08368/) (2022)
- [ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization](https://mlsanity.com/paper/2208.14286/) (2022)
- [Q-ViT: Fully Differentiable Quantization for Vision Transformer](https://mlsanity.com/paper/2201.07703/) (2022)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [SDQ: Stochastic Differentiable Quantization with Mixed Precision](https://mlsanity.com/paper/2206.04459/) (2022)
- [CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution](https://mlsanity.com/paper/2207.10345/) (2022)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Instance-Aware Group Quantization for Vision Transformers](https://mlsanity.com/paper/2404.00928/) (2024)
- [Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks](https://mlsanity.com/paper/2210.16103/) (2022)
- [Bit-Mixer: Mixed-precision networks with runtime bit-width selection](https://mlsanity.com/paper/2103.17267/) (2021)
- [EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization](https://mlsanity.com/paper/2307.10554/) (2023)
- [Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution](https://mlsanity.com/paper/2106.02295/) (2021)
- [Wavelet Feature Maps Compression for Image-to-Image CNNs](https://mlsanity.com/paper/2205.12268/) (2022)
- [M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type](https://mlsanity.com/paper/2502.18755/) (2025)
- [Q-PPG: Energy-Efficient PPG-based Heart Rate Monitoring on Wearable Devices](https://mlsanity.com/paper/2203.14907/) (2022)
- [Retraining-free Model Quantization via One-Shot Weight-Coupling Learning](https://mlsanity.com/paper/2401.01543/) (2024)
- [Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators](https://mlsanity.com/paper/2106.06575/) (2021)
- [Sparse PointPillars: Maintaining and Exploiting Input Sparsity to Improve Runtime on Embedded Systems](https://mlsanity.com/paper/2106.06882/) (2021)
- [Generalizable Mixed-Precision Quantization via Attribution Rank Preservation](https://mlsanity.com/paper/2108.02720/) (2021)
- [Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices](https://mlsanity.com/paper/2311.18129/) (2023)
- [BiDM: Pushing the Limit of Quantization for Diffusion Models](https://mlsanity.com/paper/2412.05926/) (2024)
- [EQ-Net: Elastic Quantization Neural Networks](https://mlsanity.com/paper/2308.07650/) (2023)
- [MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2401.14895/) (2024)
- [TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models](https://mlsanity.com/paper/2404.09532/) (2024)
- [Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes](https://mlsanity.com/paper/2206.08852/) (2022)
- [PLiNIO: A User-Friendly Library of Gradient-based Methods for Complexity-aware DNN Optimization](https://mlsanity.com/paper/2307.09488/) (2023)
- [RefQSR: Reference-based Quantization for Image Super-Resolution Networks](https://mlsanity.com/paper/2404.01690/) (2024)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)

## Abstract (arXiv)

Low-precision networks, with weights and activations quantized to low bit-width, are widely used to accelerate inference on edge devices. However, current solutions are uniform, using identical bit-width for all filters. This fails to account for the different sensitivities of different filters and is suboptimal. Mixed-precision networks address this problem, by tuning the bit-width to individual filter requirements. In this work, the problem of optimal mixed-precision network search (MPS) is considered. To circumvent its difficulties of discrete search space and combinatorial optimization, a new differentiable search architecture is proposed, with several novel contributions to advance the efficiency by leveraging the unique properties of the MPS problem. The resulting Efficient differentiable MIxed-Precision network Search (EdMIPS) method is effective at finding the optimal bit allocation for multiple popular networks, and can search a large model, e.g. Inception-V3, directly on ImageNet without proxy task in a reasonable amount of time. The learned mixed-precision networks significantly outperform their uniform counterparts.