# Rethinking floating point for deep learning

arXiv: https://arxiv.org/abs/1811.01721 · HTML: https://mlsanity.com/paper/1811.01721/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 22 of 22)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)
- [The Efficiency Misnomer](https://mlsanity.com/paper/2110.12894/) (2021)
- [ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity](https://mlsanity.com/paper/2203.08101/) (2022)
- [8-bit Numerical Formats for Deep Neural Networks](https://mlsanity.com/paper/2206.02915/) (2022)
- [Single-Shot Optical Neural Network](https://mlsanity.com/paper/2205.09103/) (2022)
- [LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update](https://mlsanity.com/paper/2106.13914/) (2021)
- [Algorithm-Hardware Co-Design of Distribution-Aware Logarithmic-Posit Encodings for Efficient DNN Inference](https://mlsanity.com/paper/2403.05465/) (2024)
- [Multiplication-Free Transformer Training via Piecewise Affine Operations](https://mlsanity.com/paper/2305.17190/) (2023)
- [Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators](https://mlsanity.com/paper/2101.11748/) (2021)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [PLAM: a Posit Logarithm-Approximate Multiplier](https://mlsanity.com/paper/2102.09262/) (2021)
- [MinUn: Accurate ML Inference on Microcontrollers](https://mlsanity.com/paper/2210.16556/) (2022)
- [PositNN: Training Deep Neural Networks with Mixed Low-Precision Posit](https://mlsanity.com/paper/2105.00053/) (2021)
- [FPGA-Accelerated SpeckleNN with SNL for Real-time X-ray Single-Particle Imaging](https://mlsanity.com/paper/2502.19734/) (2025)
- [LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training](https://mlsanity.com/paper/2211.02686/) (2022)
- [Defensive ML: Defending Architectural Side-channels with Adversarial Obfuscation](https://mlsanity.com/paper/2302.01474/) (2023)
- [Compressed Real Numbers for AI: a case-study using a RISC-V CPU](https://mlsanity.com/paper/2309.07158/) (2023)
- [An Open-Source Framework for Efficient Numerically-Tailored Computations](https://mlsanity.com/paper/2406.02579/) (2024)
- [Investigating Long-term Training for Remote Sensing Object Detection](https://mlsanity.com/paper/2407.15143/) (2024)
- [Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation](https://mlsanity.com/paper/2602.14701/) (2026)
- [Chameleon: Dynamic Format Adapter for Efficient Diffusion](https://mlsanity.com/paper/2609.33496/) (2026)

## Abstract (arXiv)

Reducing hardware overhead of neural networks for faster or lower power inference and training is an active area of research. Uniform quantization using integer multiply-add has been thoroughly investigated, which requires learning many quantization parameters, fine-tuning training or other prerequisites. Little effort is made to improve floating point relative to this baseline; it remains energy inefficient, and word size reduction yields drastic loss in needed dynamic range. We improve floating point to be more energy efficient than equivalent bit width integer hardware on a 28 nm ASIC process while retaining accuracy in 8 bits with a novel hybrid log multiply/linear add, Kulisch accumulation and tapered encodings from Gustafson's posit format. With no network retraining, and drop-in replacement of all math and float32 parameters via round-to-nearest-even only, this open-sourced 8-bit log float is within 0.9% top-1 and 0.2% top-5 accuracy of the original float32 ResNet-50 CNN model on ImageNet. Unlike int8 quantization, it is still a general purpose floating point arithmetic, interpretable out-of-the-box. Our 8/38-bit log float multiply-add is synthesized and power profiled at 28 nm at 0.96x the power and 1.12x the area of 8/32-bit integer multiply-add. In 16 bits, our log float multiply-add is 0.59x the power and 0.68x the area of IEEE 754 float16 fused multiply-add, maintaining the same signficand precision and dynamic range, proving useful for training ASICs as well.