mlsanityData to 14 Oct 2026

TEQ: Trainable Equivalent Transformation for Quantization of LLMs

arXiv 2310.10944 ↗First posted 17 Oct 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) become more prevalent, there is a growing need for new and improved quantization methods that can meet the computationalast layer demands of these modern architectures while maintaining the accuracy. In this paper, we present TEQ, a trainable equivalent transformation that preserves the FP32 precision of the model output while taking advantage of low-precision quantization, especially 3 and 4 bits weight-only quantization. The training process is lightweight, requiring only 1K steps and fewer than 0.1 percent of the original model's trainable parameters. Furthermore, the transformation does not add any computational overhead during inference. Our results are on-par with the state-of-the-art (SOTA) methods on typical LLMs. Our approach can be combined with other methods to achieve even better performance. The code is available at https://github.com/intel/neural-compressor.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

28 of 40 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Layer Normalizationcited by 5,669
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
16 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  7. 2019Learned Step Size Quantizationcited by 628
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  11. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  12. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  13. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  14. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  15. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26
  16. 2019Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activationscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 1.

9 papers.

  1. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  2. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  3. 2023Efficient LLM Inference on CPUscited by 24
  4. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 22
  5. 2024QEFT: Quantization for Efficient Fine-Tuning of LLMscited by 7
  6. 2025Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Modelscited by 2
  7. 2024ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNscited by 0
  8. 2025SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMscited by 0
  9. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  2. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
  3. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  4. 2026SliderQuant: Accurate Post-Training Quantization for LLMs
  5. 2024Low-Rank Quantization-Aware Training for LLMs
  6. 2026REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
  7. 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
  8. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices