mlsanityData to 14 Oct 2026

Dynamic Stashing Quantization for Efficient Transformer Training

arXiv 2303.05295 ↗First posted 9 Mar 2023cs.LG cs.CL cs.PFcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive performance on a range of Natural Language Processing (NLP) tasks. Unfortunately, the immense amount of computations and memory accesses required for LLM training makes them prohibitively expensive in terms of hardware cost, and thus challenging to deploy in use cases such as on-device learning. In this paper, motivated by the observation that LLM training is memory-bound, we propose a novel dynamic quantization strategy, termed Dynamic Stashing Quantization (DSQ), that puts a special focus on reducing the memory operations, but also enjoys the other benefits of low precision training, such as the reduced arithmetic cost. We conduct a thorough study on two translation tasks (trained-from-scratch) and three classification tasks (fine-tuning). DSQ reduces the amount of arithmetic operations by 20.95× and the number of DRAM operations by 2.55× on IWSLT17 compared to the standard 16-bit fixed-point, which is widely used in on-device learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

18 of 31 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  8. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  9. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  10. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  11. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  12. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
6 more
  1. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 59
  2. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
  3. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 41
  4. 2018Training DNNs with Hybrid Block Floating Pointcited by 40
  5. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  6. 2020FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Trainingcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 1; 2026: 1.

6 papers.

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  3. 2023Hadamard Domain Training with Integers for Class Incremental Quantized Learningcited by 3
  4. 2023Understanding the Impact of Post-Training Quantization on Large Language Modelscited by 2
  5. 2025CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devicescited by 0
  6. 2026Zero-shot adaptable task planning for autonomous construction robots: a comparative study of lightweight single and multi-AI agent systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Study on Quantization Techniques for Large Language Models
  2. 2023Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
  3. 2025Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
  4. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantization
  5. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  6. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
  7. 2022Zero-Shot Dynamic Quantization for Transformer Inference
  8. 2026TernaryLM: Memory-Efficient Language Modeling via Native 1.5-Bit Quantization with Adaptive Layer-wise Scaling