# PROFIT: A Novel Training Method for sub-4-bit MobileNet Models

arXiv: https://arxiv.org/abs/2008.04693 · HTML: https://mlsanity.com/paper/2008.04693/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2020.

## Built on by (most cited first, 30 of 50)

- [Overcoming Oscillations in Quantization-Aware Training](https://mlsanity.com/paper/2203.11086/) (2022)
- [Network Quantization with Element-wise Gradient Scaling](https://mlsanity.com/paper/2104.00903/) (2021)
- [Temporal Dynamic Quantization for Diffusion Models](https://mlsanity.com/paper/2306.02316/) (2023)
- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance](https://mlsanity.com/paper/2203.08368/) (2022)
- [NIPQ: Noise proxy-based Integrated Pseudo-Quantization](https://mlsanity.com/paper/2206.00820/) (2022)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)
- [Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training](https://mlsanity.com/paper/2206.06501/) (2022)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Instance-Aware Group Quantization for Vision Transformers](https://mlsanity.com/paper/2404.00928/) (2024)
- [Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution](https://mlsanity.com/paper/2106.02295/) (2021)
- [CPT: Efficient Deep Neural Network Training via Cyclic Precision](https://mlsanity.com/paper/2101.09868/) (2021)
- [Distance-aware Quantization](https://mlsanity.com/paper/2108.06983/) (2021)
- [ESPACE: Dimensionality Reduction of Activations for Model Compression](https://mlsanity.com/paper/2410.05437/) (2024)
- [Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices](https://mlsanity.com/paper/2206.10844/) (2022)
- [PDP: Parameter-free Differentiable Pruning is All You Need](https://mlsanity.com/paper/2305.11203/) (2023)
- [MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization](https://mlsanity.com/paper/2311.06798/) (2023)
- [SEAM: Searching Transferable Mixed-Precision Quantization Policy through Large Margin Regularization](https://mlsanity.com/paper/2302.06845/) (2023)
- [Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart](https://mlsanity.com/paper/2412.15846/) (2024)
- [LOTION: Smoothing the Optimization Landscape for Quantized Training](https://mlsanity.com/paper/2510.08757/) (2025)
- [2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency](https://mlsanity.com/paper/2109.05223/) (2021)
- [Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks](https://mlsanity.com/paper/2206.07741/) (2022)
- [PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks](https://mlsanity.com/paper/2404.00103/) (2024)
- [Edge AI-Enabled Chicken Health Detection Based on Enhanced FCOS-Lite and Knowledge Distillation](https://mlsanity.com/paper/2407.09562/) (2024)
- [LG-LSQ: Learned Gradient Linear Symmetric Quantization](https://mlsanity.com/paper/2202.09009/) (2022)
- [AMED: Automatic Mixed-Precision Quantization for Edge Devices](https://mlsanity.com/paper/2205.15437/) (2022)
- [Symmetry Regularization and Saturating Nonlinearity for Robust Quantization](https://mlsanity.com/paper/2208.00338/) (2022)
- [Mixed Precision Post Training Quantization of Neural Networks with Sensitivity Guided Search](https://mlsanity.com/paper/2302.01382/) (2023)
- [Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantization](https://mlsanity.com/paper/2306.04879/) (2023)
- [Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer](https://mlsanity.com/paper/2104.00210/) (2021)

## Abstract (arXiv)

4-bit and lower precision mobile models are required due to the ever-increasing demand for better energy efficiency in mobile devices. In this work, we report that the activation instability induced by weight quantization (AIWQ) is the key obstacle to sub-4-bit quantization of mobile networks. To alleviate the AIWQ problem, we propose a novel training method called PROgressive-Freezing Iterative Training (PROFIT), which attempts to freeze layers whose weights are affected by the instability problem stronger than the other layers. We also propose a differentiable and unified quantization method (DuQ) and a negative padding idea to support asymmetric activation functions such as h-swish. We evaluate the proposed methods by quantizing MobileNet-v1, v2, and v3 on ImageNet and report that 4-bit quantization offers comparable (within 1.48 % top-1 accuracy) accuracy to full precision baseline. In the ablation study of the 3-bit quantization of MobileNet-v3, our proposed method outperforms the state-of-the-art method by a large margin, 12.86 % of top-1 accuracy.