mlsanityData to 14 Oct 2026

Magic for the Age of Quantized DNNs

arXiv 2403.14999 ↗First posted 22 Mar 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recently, the number of parameters in DNNs has explosively increased, as exemplified by LLMs (Large Language Models), making inference on small-scale computers more difficult. Model compression technology is, therefore, essential for integration into products. In this paper, we propose a method of quantization-aware training. We introduce a novel normalization (Layer-Batch Normalization) that is independent of the mini-batch size and does not require any additional computation cost during inference. Then, we quantize the weights by the scaled round-clip function with the weight standardization. We also quantize activation functions using the same function and apply surrogate gradients to train the model with both quantized weights and the quantized activation functions. We call this method Magic for the age of Quantised DNNs (MaQD). Experimental results show that our quantization method can be achieved with minimal accuracy degradation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

20 of 35 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  9. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  10. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  11. 2018Model compression via distillation and quantizationcited by 289
  12. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
8 more
  1. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  2. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  3. 2022On-Device Training Under 256KB Memorycited by 160
  4. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 105
  5. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 89
  6. 2022Online Training Through Time for Spiking Neural Networkscited by 49
  7. 2019Micro-Batch Training with Batch-Channel Normalization and Weight Standardizationcited by 46
  8. 2023Spike Accumulation Forwarding for Effective Training of Spiking Neural Networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Low-bit Model Quantization for Deep Neural Networks: A Survey
  2. 2026Quantization-Aware Regularizers for Deep Neural Networks Compression
  3. 2024QGen: On the Ability to Generalize in Quantization Aware Training
  4. 2025Towards Superior Quantization Accuracy: A Layer-sensitive Approach
  5. 2021Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
  6. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview
  7. 2022Post-Training Quantization for Energy Efficient Realization of Deep Neural Networks
  8. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs