# Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks

arXiv: https://arxiv.org/abs/2010.15703 · HTML: https://mlsanity.com/paper/2010.15703/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 28 of 28)

- [GPTVQ: The Blessing of Dimensionality for LLM Quantization](https://mlsanity.com/paper/2402.15319/) (2024)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- [An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2303.00633/) (2023)
- [Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling](https://mlsanity.com/paper/2405.13796/) (2024)
- [VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformers](https://mlsanity.com/paper/2408.17131/) (2024)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [Fine-grained Data Distribution Alignment for Post-Training Quantization](https://mlsanity.com/paper/2109.04186/) (2021)
- [Enabling On-Device Smartphone GPU based Training: Lessons Learned](https://mlsanity.com/paper/2202.10100/) (2022)
- [YONO: Modeling Multiple Heterogeneous Neural Networks on Microcontrollers](https://mlsanity.com/paper/2203.03794/) (2022)
- [Sub-8-bit quantization for on-device speech recognition: a regularization-free approach](https://mlsanity.com/paper/2210.09188/) (2022)
- [Hyperspherical Quantization: Toward Smaller and More Accurate Models](https://mlsanity.com/paper/2212.12653/) (2022)
- [Quantized Sparse Weight Decomposition for Neural Network Compression](https://mlsanity.com/paper/2207.11048/) (2022)
- [Multi-modal Streaming 3D Object Detection](https://mlsanity.com/paper/2209.04966/) (2022)
- [ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba](https://mlsanity.com/paper/2503.09509/) (2025)
- [Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs](https://mlsanity.com/paper/2608.04048/) (2026)
- [CPrune: Compiler-Informed Model Pruning for Efficient Target-Aware DNN Execution](https://mlsanity.com/paper/2207.01260/) (2022)
- [Learning Low-Rank Representations for Model Compression](https://mlsanity.com/paper/2211.11397/) (2022)
- [MVQ:Towards Efficient DNN Compression and Acceleration with Masked Vector Quantization](https://mlsanity.com/paper/2412.10261/) (2024)
- [LegoNet: Memory Footprint Reduction Through Block Weight Clustering](https://mlsanity.com/paper/2603.06606/) (2026)
- [Pruning Ternary Quantization](https://mlsanity.com/paper/2107.10998/) (2021)
- [Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression](https://mlsanity.com/paper/2111.10320/) (2021)
- [Deep learning model compression using network sensitivity and gradients](https://mlsanity.com/paper/2210.05111/) (2022)
- [Hyperspherical Loss-Aware Ternary Quantization](https://mlsanity.com/paper/2212.12649/) (2022)
- [Mobile Fitting Room: On-device Virtual Try-on via Diffusion Models](https://mlsanity.com/paper/2402.01877/) (2024)
- [D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations](https://mlsanity.com/paper/2403.19163/) (2024)
- [A Novel Structure-Agnostic Multi-Objective Approach for Weight-Sharing Compression in Deep Neural Networks](https://mlsanity.com/paper/2501.03095/) (2025)
- [SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splitting](https://mlsanity.com/paper/2503.08668/) (2025)
- [A Theory of Machine Understanding via the Minimum Description Length Principle](https://mlsanity.com/paper/2504.00395/) (2025)

## Abstract (arXiv)

Compressing large neural networks is an important step for their deployment in resource-constrained computational platforms. In this context, vector quantization is an appealing framework that expresses multiple parameters using a single code, and has recently achieved state-of-the-art network compression on a range of core vision and natural language processing tasks. Key to the success of vector quantization is deciding which parameter groups should be compressed together. Previous work has relied on heuristics that group the spatial dimension of individual convolutional filters, but a general solution remains unaddressed. This is desirable for pointwise convolutions (which dominate modern architectures), linear layers (which have no notion of spatial dimension), and convolutions (when more than one filter is compressed to the same codeword). In this paper we make the observation that the weights of two adjacent layers can be permuted while expressing the same function. We then establish a connection to rate-distortion theory and search for permutations that result in networks that are easier to compress. Finally, we rely on an annealed quantization algorithm to better compress the network and achieve higher final accuracy. We show results on image classification, object detection, and segmentation, reducing the gap with the uncompressed model by 40 to 70% with respect to the current state of the art.