mlsanityData to 7 Oct 2026

RPTQ: Reorder-based Post-training Quantization for Large Language Models

arXiv 2304.01089 ↗First posted 3 Apr 2023cs.CLcited by 112

Not on the map. Why?

Abstract (arXiv)

Large-scale language models (LLMs) have demonstrated impressive performance, but their deployment presents challenges due to their significant memory usage. This issue can be alleviated through quantization. In this paper, we identify that the challenge in quantizing activations in LLMs arises from varying ranges across channels, rather than solely the presence of outliers. To address this challenge, we introduce a quantization method called RPTQ, which utilizes a reorder-based approach. By rearranging the channels and quantizing them in clusters, RPTQ effectively mitigates the impact of range differences between channels. To minimize the overhead of the reorder operation, we fuse it into the layer norm operation and weights in linear layers. In our experiments, RPTQ achieved a significant breakthrough by utilizing 3-bit activation in LLMs for the first time, resulting in a substantial reduction in memory usage. For instance, quantizing OPT-175b can lead to a memory consumption reduction of up to 80%.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

35 of 44 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
23 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2019Learned Step Size Quantizationcited by 604
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  13. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  15. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  16. 2022Large Language Models with Controllable Working Memorycited by 169
  17. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  18. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  19. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  20. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  21. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  22. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  23. 2021PTQ-SL: Exploring the Sub-layerwise Post-training Quantizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 40; 2025: 25; 2026: 23.

112 papers.

Sort:

By area:

  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  2. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  3. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  4. 2023A Survey on Model Compression for Large Language Modelscited by 263
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  6. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  9. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
38 more
  1. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  2. 2024Evaluating Quantized Large Language Modelscited by 75
  3. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  4. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 64
  5. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  7. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  8. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 49
  9. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  10. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  11. 2024A Survey on Transformer Compressioncited by 44
  12. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  13. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
  14. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  15. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  16. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  17. 2023CBQ: Cross-Block Quantization for Large Language Modelscited by 34
  18. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  19. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  20. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  21. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spikingcited by 29
  22. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  23. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  24. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23
  25. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  26. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 22
  27. 2025An Empirical Study of Qwen3 Quantizationcited by 20
  28. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 19
  29. 2025Towards AI Search Paradigmcited by 19
  30. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  31. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  32. 2024LLMCBench: Benchmarking Large Language Model Compression for Efficient Deploymentcited by 16
  33. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  34. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  35. 2024CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulnesscited by 12
  36. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  37. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  38. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterizationcited by 10

50 most cited of 112 All 112 (JSON)