# OMPQ: Orthogonal Mixed Precision Quantization

arXiv: https://arxiv.org/abs/2109.07865 · HTML: https://mlsanity.com/paper/2109.07865/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 58 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Deep Learning with Low Precision by Half-wave Gaussian Quantization](https://mlsanity.com/paper/1702.00953/) (2017)
- [FracBits: Mixed Precision Quantization via Fractional Bit-Widths](https://mlsanity.com/paper/2007.02017/) (2020)
- [Value-aware Quantization for Training and Inference of Neural Networks](https://mlsanity.com/paper/1804.07802/) (2018)
- [PAMS: Quantized Super-Resolution via Parameterized Max Scale](https://mlsanity.com/paper/2011.04212/) (2020)
- [Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagation](https://mlsanity.com/paper/1910.10853/) (2019)

## Built on by (most cited first, 30 of 39)

- [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2405.14917/) (2024)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [SDQ: Stochastic Differentiable Quantization with Mixed Precision](https://mlsanity.com/paper/2206.04459/) (2022)
- [A Practical Mixed Precision Algorithm for Post-Training Quantization](https://mlsanity.com/paper/2302.05397/) (2023)
- [EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization](https://mlsanity.com/paper/2307.10554/) (2023)
- [Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective](https://mlsanity.com/paper/2303.11906/) (2023)
- [Quantized Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2509.21302/) (2025)
- [Patch-wise Mixed-Precision Quantization of Vision Transformer](https://mlsanity.com/paper/2305.06559/) (2023)
- [SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models](https://mlsanity.com/paper/2603.16859/) (2026)
- [MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation](https://mlsanity.com/paper/2507.04290/) (2025)
- [Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices](https://mlsanity.com/paper/2311.18129/) (2023)
- [Channel-Wise Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2410.13056/) (2024)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization](https://mlsanity.com/paper/2604.17789/) (2026)
- [ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models](https://mlsanity.com/paper/2605.24011/) (2026)
- [Mixed-Precision Quantization for Deep Vision Models with Integer Quadratic Programming](https://mlsanity.com/paper/2307.05657/) (2023)
- [Motion-Aware Caching for Efficient Autoregressive Video Generation](https://mlsanity.com/paper/2605.01725/) (2026)
- [ParZC: Parametric Zero-Cost Proxies for Efficient NAS](https://mlsanity.com/paper/2402.02105/) (2024)
- [Towards Optimal Compression: Joint Pruning and Quantization](https://mlsanity.com/paper/2302.07612/) (2023)
- [Lossy and Lossless (L2) Post-training Model Size Compression](https://mlsanity.com/paper/2308.04269/) (2023)
- [InfoQ: Mixed-Precision Quantization via Global Information Flow](https://mlsanity.com/paper/2508.04753/) (2025)
- [Efficient and Effective Methods for Mixed Precision Neural Network Quantization for Faster, Energy-efficient Inference](https://mlsanity.com/paper/2301.13330/) (2023)
- [MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspective](https://mlsanity.com/paper/2507.19131/) (2025)
- [On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks](https://mlsanity.com/paper/2309.01945/) (2023)
- [AdaQAT: Adaptive Bit-Width Quantization-Aware Training](https://mlsanity.com/paper/2404.16876/) (2024)
- [Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantization](https://mlsanity.com/paper/2508.03002/) (2025)
- [Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning](https://mlsanity.com/paper/2605.28160/) (2026)
- [OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models](https://mlsanity.com/paper/2607.23193/) (2026)
- [Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement](https://mlsanity.com/paper/2609.25916/) (2026)
- [A Data-Free Analytical Quantization Scheme for Deep Learning Models](https://mlsanity.com/paper/2412.07391/) (2024)

## Abstract (arXiv)

To bridge the ever increasing gap between deep neural networks' complexity and hardware capability, network quantization has attracted more and more research attention. The latest trend of mixed precision quantization takes advantage of hardware's multiple bit-width arithmetic operations to unleash the full potential of network quantization. However, this also results in a difficult integer programming formulation, and forces most existing approaches to use an extremely time-consuming search process even with various relaxations. Instead of solving a problem of the original integer programming, we propose to optimize a proxy metric, the concept of network orthogonality, which is highly correlated with the loss of the integer programming but also easy to optimize with linear programming. This approach reduces the search time and required data amount by orders of magnitude, with little compromise on quantization accuracy. Specifically, we achieve 72.08% Top-1 accuracy on ResNet-18 with 6.7Mb, which does not require any searching iterations. Given the high efficiency and low data dependency of our algorithm, we used it for the post-training quantization, which achieve 71.27% Top-1 accuracy on MobileNetV2 with only 1.5Mb. Our code is available at https://github.com/MAC-AutoML/OMPQ.