mlsanityData to 14 Oct 2026

Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators

arXiv 2401.14110 ↗First posted 25 Jan 2024cs.LG cs.AI cs.ARcited by 4

Not on the map. Why?

Abstract (arXiv)

The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision of tensors visible by high-level frameworks (e.g., weights, activations, and gradients). However, current hardware still relies on high-accuracy core operations. Most significant is the operation of accumulating products. This high-precision accumulation operation is gradually becoming the main computational bottleneck. This is because, so far, the usage of low-precision accumulators led to a significant degradation in performance. In this work, we present a simple method to train and fine-tune high-end DNNs, to allow, for the first time, utilization of cheaper, 12-bits accumulators, with no significant degradation in accuracy. Lastly, we show that as we decrease the accumulation precision further, using fine-grained gradient approximations can improve the DNN accuracy.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

20 of 28 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  10. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  11. 2015Deep Learning with Limited Numerical Precisioncited by 376
  12. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
8 more
  1. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  2. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  3. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  4. 2022FP8 Quantization: The Power of the Exponentcited by 88
  5. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  6. 2019QPyTorch: A Low-Precision Arithmetic Simulation Frameworkcited by 22
  7. 2020WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmeticcited by 12
  8. 2019Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networkscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Accumulator-Aware Post-Training Quantization for Large Language Modelscited by 6
  2. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  3. 2025Exploring the Performance Improvement of Tensor Processing Engines through Transformation in the Bit-weight Dimension of MACscited by 0
  4. 2025PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance
  2. 2023A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance
  3. 2024A2Q+: Improving Accumulator-Aware Weight Quantization
  4. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
  5. 2022Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks
  6. 2025PQS (Prune, Quantize, and Sort): Low-Bitwidth Accumulation of Dot Products in Neural Network Computations
  7. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
  8. 2025Low-bit Model Quantization for Deep Neural Networks: A Survey