mlsanityData to 7 Oct 2026

The Hidden Power of Pure 16-bit Floating-Point Neural Networks

arXiv 2301.12809 ↗First posted 30 Jan 2023cs.LG cs.AI cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Lowering the precision of neural networks from the prevalent 32-bit precision has long been considered harmful to performance, despite the gain in space and time. Many works propose various techniques to implement half-precision neural networks, but none study pure 16-bit settings. This paper investigates the unexpected performance gain of pure 16-bit neural networks over the 32-bit networks in classification tasks. We present extensive experimental results that favorably compare various 16-bit neural networks' performance to those of the 32-bit models. In addition, a theoretical analysis of the efficiency of 16-bit models is provided, which is coupled with empirical evidence to back it up. Finally, we discuss situations in which low-precision training is indeed detrimental.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

16 of 28 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2017Mixed Precision Trainingcited by 1,135
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2015Deep Learning with Limited Numerical Precisioncited by 368
  10. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  11. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  12. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 95
4 more
  1. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  2. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  3. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 43
  4. 2019The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Designcited by 15

Built on by

Not cited yet.