mlsanityData to 14 Oct 2026

Number Systems for Deep Neural Network Architectures: A Survey

arXiv 2307.05035 ↗First posted 11 Jul 2023cs.NE cs.AR cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) have become an enabling component for a myriad of artificial intelligence applications. DNNs have shown sometimes superior performance, even compared to humans, in cases such as self-driving, health applications, etc. Because of their computational complexity, deploying DNNs in resource-constrained devices still faces many challenges related to computing complexity, energy efficiency, latency, and cost. To this end, several research directions are being pursued by both academia and industry to accelerate and efficiently implement DNNs. One important direction is determining the appropriate data representation for the massive amount of data involved in DNN processing. Using conventional number systems has been found to be sub-optimal for DNNs. Alternatively, a great body of research focuses on exploring suitable number systems. This article aims to provide a comprehensive survey and discussion about alternative number systems for more efficient representations of DNN data. Various number systems (conventional/unconventional) exploited for DNNs are discussed. The impact of these number systems on the performance and hardware design of DNNs is considered. In addition, this paper highlights the challenges associated with each number system and various solutions that are proposed for addressing them. The reader will be able to understand the importance of an efficient number system for DNN, learn about the widely used number systems for DNN, understand the trade-offs between various number systems, and consider various design aspects that affect the impact of number systems on DNN performance. In addition, the recent trends and related research opportunities will be highlighted

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 44 of its 170 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

44 of 170 references.

References · 44

  1. 2017Mixed Precision Trainingcited by 1,156
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  4. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  5. 2015Deep Learning with Limited Numerical Precisioncited by 376
  6. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  7. 2020Binary Neural Networks: A Surveycited by 190
  8. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 143
  9. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 99
  10. 2019Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Surveycited by 93
  11. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  12. 2015Neural Networks with Few Multiplicationscited by 48
32 more
  1. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  2. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 44
  3. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
  4. 2018Training DNNs with Hybrid Block Floating Pointcited by 40
  5. 2019Survey on Deep Neural Networks in Speech and Vision Systemscited by 37
  6. 2021FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Roundingcited by 31
  7. 2018Rethinking floating point for deep learningcited by 22
  8. 2016Hardware-oriented Approximation of Convolutional Neural Networkscited by 20
  9. 2017An OpenCL(TM) Deep Learning Accelerator on Arria 10cited by 13
  10. 2018Deep Positron: A Deep Neural Network Using the Posit Number Systemcited by 13
  11. 2020Model Fusion with Kullback--Leibler Divergencecited by 10
  12. 2022FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Supportcited by 10
  13. 2016Overcoming Challenges in Fixed Point Training of Deep Convolutional Networkscited by 9
  14. 2017Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activationscited by 7
  15. 2017Computation Error Analysis of Block Floating Point Arithmetic Oriented Convolution Neural Network Accelerator Designcited by 7
  16. 2019Performance-Efficiency Trade-off of Low-Precision Numerical Formats in Deep Neural Networkscited by 7
  17. 2019Neural Network Training with Approximate Logarithmic Computationscited by 7
  18. 2021LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Updatecited by 7
  19. 2020Low Precision Floating-point Arithmetic for High Performance FPGA-based CNN Accelerationcited by 5
  20. 2021FIXAR: A Fixed-Point Deep Reinforcement Learning Platform with Quantization-Aware Training and Adaptive Parallelismcited by 5
  21. 2021Fixed-Posit: A Floating-Point Representation for Error-Resilient Applicationscited by 5
  22. 2017Mixed Low-precision Deep Learning Inference using Dynamic Fixed Pointcited by 4
  23. 2018Deep Learning Inference on Embedded Devices: Fixed-Point vs Positcited by 4
  24. 2021Rethinking Floating Point Overheads for Mixed Precision DNN Acceleratorscited by 4
  25. 2020ExPAN(N)D: Exploring Posits for Efficient Artificial Neural Network Design in FPGA-based Systemscited by 3
  26. 2021PLAM: a Posit Logarithm-Approximate Multipliercited by 3
  27. 2016Multiplier-less Artificial Neurons Exploiting Error Resiliency for Energy-Efficient Neural Computingcited by 2
  28. 2019Template-Based Posit Multiplication for Training and Inferring in Neural Networkscited by 2
  29. 2022FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representationcited by 2
  30. 2017Applying the Residue Number System to Network Inferencecited by 1
  31. 2018QUENN: QUantization Engine for low-power Neural Networkscited by 1
  32. 2020A Very Compact Embedded CNN Processor Design Based on Logarithmic Computingcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 28
  2. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  3. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applicationscited by 7
  4. 2025Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computationcited by 0