mlsanityData to 7 Oct 2026

Deep Learning with Limited Numerical Precision

arXiv 1502.02551 ↗First posted 9 Feb 2015cs.LG cs.NE stat.MLcited by 368

Not on the map. Why?

Abstract (arXiv)

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

Key facts. Cited by 368 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 90; 2022: 74; 2023: 66; 2024: 55; 2025: 45; 2026: 37.

368 papers.

Sort:

By area:

  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2021A White Paper on Neural Network Quantizationcited by 453
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2022Teaching language models to support answers with verified quotescited by 290
  7. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  8. 2021NeRV: Neural Representations for Videoscited by 255
  9. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  10. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 166
  11. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  12. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
38 more
  1. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  2. 2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 127
  3. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  4. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  5. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  6. 2023HNeRV: A Hybrid Neural Representation for Videoscited by 100
  7. 2022FP8 Quantization: The Power of the Exponentcited by 87
  8. 2021Vision Transformer Pruningcited by 77
  9. 2022Deep Differentiable Logic Gate Networkscited by 67
  10. 2022Compressing Volumetric Radiance Fields to 1 MBcited by 66
  11. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  12. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 61
  13. 2021A Comprehensive Survey on Hardware-Aware Neural Architecture Searchcited by 60
  14. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  15. 2021A Survey on Green Deep Learningcited by 52
  16. 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 51
  17. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  18. 2023Intriguing Properties of Quantization at Scalecited by 48
  19. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  20. 2023Pruning vs Quantization: Which is Better?cited by 45
  21. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  22. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  23. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  24. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  25. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 40
  26. 2022Hardware Approximate Techniques for Deep Neural Network Accelerators: A Surveycited by 39
  27. 2023A Guide Through the Zoo of Biased SGDcited by 39
  28. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  29. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  30. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  31. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  32. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  33. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPscited by 34
  34. 2021Q-SpiNN: A Framework for Quantizing Spiking Neural Networkscited by 33
  35. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  36. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  37. 2021FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Roundingcited by 31
  38. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31

50 most cited of 368 All 368 (JSON)