# Ternary Neural Networks with Fine-Grained Quantization

arXiv: https://arxiv.org/abs/1705.01462 · HTML: https://mlsanity.com/paper/1705.01462/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2017.

## Built on by (most cited first, 17 of 17)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats](https://mlsanity.com/paper/2307.09782/) (2023)
- [Efficient Post-training Quantization with FP8 Formats](https://mlsanity.com/paper/2309.14592/) (2023)
- [Bit-Mixer: Mixed-precision networks with runtime bit-width selection](https://mlsanity.com/paper/2103.17267/) (2021)
- [Extreme Compression for Pre-trained Transformers Made Simple and Efficient](https://mlsanity.com/paper/2206.01859/) (2022)
- [Q-LIC: Quantizing Learned Image Compression with Channel Splitting](https://mlsanity.com/paper/2205.14510/) (2022)
- [Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training?](https://mlsanity.com/paper/2212.01076/) (2022)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [End-to-End Learned Image Compression with Quantized Weights and Activations](https://mlsanity.com/paper/2111.09348/) (2021)
- [FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation](https://mlsanity.com/paper/2203.12091/) (2022)
- [Ternary Quantization: A Survey](https://mlsanity.com/paper/2303.01505/) (2023)
- [RCT: Resource Constrained Training for Edge AI](https://mlsanity.com/paper/2103.14493/) (2021)
- [CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method](https://mlsanity.com/paper/2110.02550/) (2021)
- [Computer Vision Model Compression Techniques for Embedded Systems: A Survey](https://mlsanity.com/paper/2408.08250/) (2024)
- [A High-Performance Accelerator for Super-Resolution Processing on Embedded GPU](https://mlsanity.com/paper/2303.08999/) (2023)
- [Quality Scalable Quantization Methodology for Deep Learning on Edge](https://mlsanity.com/paper/2407.11260/) (2024)

## Abstract (arXiv)

We propose a novel fine-grained quantization (FGQ) method to ternarize pre-trained full precision models, while also constraining activations to 8 and 4-bits. Using this method, we demonstrate a minimal loss in classification accuracy on state-of-the-art topologies without additional training. We provide an improved theoretical formulation that forms the basis for a higher quality solution using FGQ. Our method involves ternarizing the original weight tensor in groups of N weights. Using N=4, we achieve Top-1 accuracy within 3.7% and 4.2% of the baseline full precision result for Resnet-101 and Resnet-50 respectively, while eliminating 75% of all multiplications. These results enable a full 8/4-bit inference pipeline, with best-reported accuracy using ternary weights on ImageNet dataset, with a potential of 9× improvement in performance. Also, for smaller networks like AlexNet, FGQ achieves state-of-the-art results. We further study the impact of group size on both performance and accuracy. With a group size of N=64, we eliminate ≈99% of the multiplications; however, this introduces a noticeable drop in accuracy, which necessitates fine tuning the parameters at lower precision. We address this by fine-tuning Resnet-50 with 8-bit activations and ternary weights at N=64, improving the Top-1 accuracy to within 4% of the full precision result with <30% additional training overhead. Our final quantized model can run on a full 8-bit compute pipeline using 2-bit weights and has the potential of up to 15× improvement in performance compared to baseline full-precision models.