# Learning Discrete Weights Using the Local Reparameterization Trick

arXiv: https://arxiv.org/abs/1710.07739 · HTML: https://mlsanity.com/paper/1710.07739/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2017.

## Built on by (most cited first, 29 of 29)

- [Differentiable Model Compression via Pseudo Quantization Noise](https://mlsanity.com/paper/2104.09987/) (2021)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)
- [SiMaN: Sign-to-Magnitude Network Binarization](https://mlsanity.com/paper/2102.07981/) (2021)
- [Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss](https://mlsanity.com/paper/2109.02100/) (2021)
- [Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling](https://mlsanity.com/paper/2205.14283/) (2022)
- [Demystifying and Generalizing BinaryConnect](https://mlsanity.com/paper/2110.13220/) (2021)
- [Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks](https://mlsanity.com/paper/2112.15139/) (2021)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)
- [Signing the Supermask: Keep, Hide, Invert](https://mlsanity.com/paper/2201.13361/) (2022)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [TyXe: Pyro-based Bayesian neural nets for Pytorch](https://mlsanity.com/paper/2110.00276/) (2021)
- [Federated Learning via Plurality Vote](https://mlsanity.com/paper/2110.02998/) (2021)
- [Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation](https://mlsanity.com/paper/2201.10360/) (2022)
- [Probabilistic Selective Encryption of Convolutional Neural Networks for Hierarchical Services](https://mlsanity.com/paper/2105.12344/) (2021)
- [DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization](https://mlsanity.com/paper/2208.09708/) (2022)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators](https://mlsanity.com/paper/2110.03549/) (2021)
- [Hyperspherical Quantization: Toward Smaller and More Accurate Models](https://mlsanity.com/paper/2212.12653/) (2022)
- [Towards Optimal Compression: Joint Pruning and Quantization](https://mlsanity.com/paper/2302.07612/) (2023)
- [Ternary Quantization: A Survey](https://mlsanity.com/paper/2303.01505/) (2023)
- [Efficient Learning of Discrete-Continuous Computation Graphs](https://mlsanity.com/paper/2307.14193/) (2023)
- [One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training](https://mlsanity.com/paper/2401.16760/) (2024)
- [Learning Disentangled Discrete Representations](https://mlsanity.com/paper/2307.14151/) (2023)
- [Pruning Ternary Quantization](https://mlsanity.com/paper/2107.10998/) (2021)
- [Nonlocal optimization of binary neural networks](https://mlsanity.com/paper/2204.01935/) (2022)
- [Learning Discrete Weights and Activations Using the Local Reparameterization Trick](https://mlsanity.com/paper/2307.01683/) (2023)
- [Efficient Model Compression for Bayesian Neural Networks](https://mlsanity.com/paper/2411.00273/) (2024)
- [SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions](https://mlsanity.com/paper/2510.08999/) (2025)
- [APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning](https://mlsanity.com/paper/2608.05499/) (2026)

## Abstract (arXiv)

Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge. One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.