mlsanityData to 7 Oct 2026

Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models

arXiv 2309.15531 ↗First posted 27 Sep 2023cs.LGcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated remarkable success across various tasks. However, efficiently serving LLMs has been a challenge due to the large memory bottleneck, specifically in small batch inference settings (e.g. mobile devices). Weight-only quantization can be a promising approach, but sub-4 bit quantization remains a challenge due to large-magnitude activation outliers. To mitigate the undesirable outlier effect, we first propose per-IC quantization, a simple yet effective method that creates quantization groups within each input channel (IC) rather than the conventional per-output-channel (per-OC). Our method is motivated by the observation that activation outliers affect the input dimension of the weight matrix, so similarly grouping the weights in the IC direction can isolate outliers within a group. We also find that activation outliers do not dictate quantization difficulty, and inherent weight sensitivities also exist. With per-IC quantization as a new outlier-friendly scheme, we propose Adaptive Dimensions (AdaDim), a versatile quantization framework that can adapt to various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim by augmenting prior methods such as Round-To-Nearest and GPTQ, showing significant improvements across various language modeling benchmarks for both base (up to +4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval) LLMs. Code is available at https://github.com/johnheo/adadim-llm

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 47 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

30 of 47 references.

References · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
18 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  10. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  11. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  13. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  14. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  15. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  16. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  17. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  18. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
  2. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization
  3. 2024GWQ: Gradient-Aware Weight Quantization for Large Language Models
  4. 2026Theory-optimal Quantization Based on Flatness
  5. 2024Mamba-PTQ: Outlier Channels in Recurrent Large Language Models
  6. 2025Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs
  7. 2023SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
  8. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?