mlsanityData to 7 Oct 2026

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

arXiv 2407.11722 ↗First posted 16 Jul 2024cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

The increasing scale of Transformer models has led to an increase in their pre-training computational requirements. While quantization has proven to be effective after pre-training and during fine-tuning, applying quantization in Transformers during pre-training has remained largely unexplored at scale for language modeling. This study aims to explore the impact of quantization for efficient pre-training of Transformers, with a focus on linear layer components. By systematically applying straightforward linear quantization to weights, activations, gradients, and optimizer states, we assess its effects on model efficiency, stability, and performance during training. By offering a comprehensive recipe of effective quantization strategies to be applied during the pre-training of Transformers, we promote high training efficiency from scratch while retaining language modeling ability. Code is available at https://github.com/chandar-lab/EfficientLLMs.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

27 of 39 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2020Efficient Transformers: A Surveycited by 784
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  12. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
15 more
  1. 2021A White Paper on Neural Network Quantizationcited by 453
  2. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  3. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  4. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  5. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  6. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  7. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  8. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  9. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  10. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 58
  11. 2023Training Transformers with 4-bit Integerscited by 58
  12. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  13. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  14. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  15. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resourcescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
  2. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  3. 2025Variance Control via Weight Rescaling in LLM Pre-trainingcited by 4
  4. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4
  5. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  6. 2025Resource-Efficient Language Models: Quantization for Fast and Accessible Inferencecited by 1
  7. 2026SPQ: An Ensemble Technique for Large Language Model Compressioncited by 0
  8. 2026Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0cited by 0
  9. 2026A Target-Centric Survey of Quantization-Aware Trainingcited by 0
  10. 2026The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamicscited by 0