# Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation

arXiv: https://arxiv.org/abs/1810.01018 · HTML: https://mlsanity.com/paper/1810.01018/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2018.

## Built on by (most cited first, 20 of 20)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [Rethinking Spatial Invariance of Convolutional Networks for Object Counting](https://mlsanity.com/paper/2206.05253/) (2022)
- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization](https://mlsanity.com/paper/2208.05163/) (2022)
- [Distribution Adaptive INT8 Quantization for Training CNNs](https://mlsanity.com/paper/2102.04782/) (2021)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Quantization and Deployment of Deep Neural Networks on Microcontrollers](https://mlsanity.com/paper/2105.13331/) (2021)
- [SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://mlsanity.com/paper/2406.03470/) (2024)
- [Quantum Neural Network Compression](https://mlsanity.com/paper/2207.01578/) (2022)
- [Neural Network Verification is a Programming Language Challenge](https://mlsanity.com/paper/2501.05867/) (2025)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Learning on Hardware: A Tutorial on Neural Network Accelerators and Co-Processors](https://mlsanity.com/paper/2104.09252/) (2021)
- [RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions](https://mlsanity.com/paper/2111.00153/) (2021)
- [Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers](https://mlsanity.com/paper/2407.18175/) (2024)
- [N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores](https://mlsanity.com/paper/2112.08193/) (2021)
- [MinUn: Accurate ML Inference on Microcontrollers](https://mlsanity.com/paper/2210.16556/) (2022)
- [Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence](https://mlsanity.com/paper/2101.09225/) (2021)
- [SupeRBNN: Randomized Binary Neural Network Using Adiabatic Superconductor Josephson Devices](https://mlsanity.com/paper/2309.12212/) (2023)
- [Ternarization of Vision Language Models for use on edge devices](https://mlsanity.com/paper/2504.06298/) (2025)

## Abstract (arXiv)

In the past years, Deep convolution neural network has achieved great success in many artificial intelligence applications. However, its enormous model size and massive computation cost have become the main obstacle for deployment of such powerful algorithm in the low power and resource-limited mobile systems. As the countermeasure to this problem, deep neural networks with ternarized weights (i.e. -1, 0, +1) have been widely explored to greatly reduce the model size and computational cost, with limited accuracy degradation. In this work, we propose a novel ternarized neural network training method which simultaneously optimizes both weights and quantizer during training, differentiating from prior works. Instead of fixed and uniform weight ternarization, we are the first to incorporate the thresholds of weight ternarization into a closed-form representation using the truncated Gaussian approximation, enabling simultaneous optimization of weights and quantizer through back-propagation training. With both of the first and last layer ternarized, the experiments on the ImageNet classification task show that our ternarized ResNet-18/34/50 only has 3.9/2.52/2.16% accuracy degradation in comparison to the full-precision counterparts.