# LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update

arXiv: https://arxiv.org/abs/2106.13914 · HTML: https://mlsanity.com/paper/2106.13914/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Training and Inference with Integers in Deep Neural Networks](https://mlsanity.com/paper/1802.04680/) (2018)
- [Towards Unified INT8 Training for Convolutional Neural Network](https://mlsanity.com/paper/1912.12607/) (2019)
- [A Statistical Framework for Low-bitwidth Training of Deep Neural Networks](https://mlsanity.com/paper/2010.14298/) (2020)
- [Rethinking floating point for deep learning](https://mlsanity.com/paper/1811.01721/) (2018)
- [High-Accuracy Low-Precision Training](https://mlsanity.com/paper/1803.03383/) (2018)
- [Learning compositional functions via multiplicative weight updates](https://mlsanity.com/paper/2006.14560/) (2020)
- [Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm](https://mlsanity.com/paper/1812.11732/) (2018)
- [Neural Network Training with Approximate Logarithmic Computations](https://mlsanity.com/paper/1910.09876/) (2019)

## Built on by (most cited first, 7 of 7)

- [Guaranteed Approximation Bounds for Mixed-Precision Neural Operators](https://mlsanity.com/paper/2307.15034/) (2023)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [Reducing the Barriers to Entry for Foundation Model Training](https://mlsanity.com/paper/2404.08811/) (2024)
- [Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks](https://mlsanity.com/paper/2506.17768/) (2025)
- [M+Adam: Low-Precision Training via Additive-Multiplicative Optimization](https://mlsanity.com/paper/2607.10611/) (2026)
- [Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training](https://mlsanity.com/paper/2510.17058/) (2025)
- [CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models](https://mlsanity.com/paper/2608.10010/) (2026)

## Abstract (arXiv)

Representing deep neural networks (DNNs) in low-precision is a promising approach to enable efficient acceleration and memory reduction. Previous methods that train DNNs in low-precision typically keep a copy of weights in high-precision during the weight updates. Directly training with low-precision weights leads to accuracy degradation due to complex interactions between the low-precision number systems and the learning algorithms. To address this issue, we develop a co-designed low-precision training framework, termed LNS-Madam, in which we jointly design a logarithmic number system (LNS) and a multiplicative weight update algorithm (Madam). We prove that LNS-Madam results in low quantization error during weight updates, leading to stable performance even if the precision is limited. We further propose a hardware design of LNS-Madam that resolves practical challenges in implementing an efficient datapath for LNS computations. Our implementation effectively reduces energy overhead incurred by LNS-to-integer conversion and partial sum accumulation. Experimental results show that LNS-Madam achieves comparable accuracy to full-precision counterparts with only 8 bits on popular computer vision and natural language tasks. Compared to FP32 and FP8, LNS-Madam reduces the energy consumption by over 90% and 55%, respectively.