mlsanityData to 13 Oct 2026

Quadapter: Adapter for GPT-2 Quantization

arXiv 2211.16912 ↗First posted 30 Nov 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Transformer language models such as GPT-2 are difficult to quantize because of outliers in activations leading to a large quantization error. To adapt to the error, one must use quantization-aware training, which entails a fine-tuning process based on the dataset and the training pipeline identical to those for the original model. Pretrained language models, however, often do not grant access to their datasets and training pipelines, forcing us to rely on arbitrary ones for fine-tuning. In that case, it is observed that quantization-aware training overfits the model to the fine-tuning data. For quantization without overfitting, we introduce a quantization adapter (Quadapter), a small set of parameters that are learned to make activations quantization-friendly by scaling them channel-wise. It keeps the model parameters unchanged. By applying our method to the challenging task of quantizing GPT-2, we demonstrate that it effectively prevents the overfitting and improves the quantization performance.

Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

23 of 26 references.

References · 23

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  3. 2016Pointer Sentinel Mixture Modelscited by 3,047
  4. 2016Learning without Forgettingcited by 2,479
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,306
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 973
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 732
  8. 2019Learned Step Size Quantizationcited by 621
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 617
  10. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 518
  11. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  12. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 333
11 more
  1. 2019Q8BERT: Quantized 8Bit BERTcited by 299
  2. 2021I-BERT: Integer-only BERT Quantizationcited by 254
  3. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  4. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  5. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  6. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  7. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  8. 2019Fully Quantized Transformer for Machine Translationcited by 36
  9. 2022Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)cited by 24
  10. 2020Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translationcited by 14
  11. 2021Unsupervised Domain Adaptation with Adaptercited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 1; 2026: 1.

10 papers.

  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
  2. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  4. 2023Intriguing Properties of Quantization at Scalecited by 49
  5. 2025HuggingGraph: Understanding the Supply Chain of LLM Ecosystemcited by 8
  6. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Modelscited by 5
  7. 2023Dual Grained Quantization: Efficient Fine-Grained Quantization for LLMcited by 2
  8. 2024Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Othercited by 2
  9. 2024Compensate Quantization Errors+: Quantized Models Are Inquisitive Learnerscited by 0
  10. 2026Recti-Q: Feature-Space Rectification for Out-of-Distribution-Robust Quantized Perception in Edge Roboticscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  2. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Models
  3. 2025GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
  4. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
  5. 2024GPTQT: Quantize Large Language Models Twice to Push the Efficiency
  6. 2026Power-of-Two Quantization-Aware-Training (PoT-QAT) in Large Language Models (LLMs)
  7. 2025QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models
  8. 2023Gradient-Based Post-Training Quantization: Challenging the Status Quo