# Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

arXiv: https://arxiv.org/abs/2311.05161 · HTML: https://mlsanity.com/paper/2311.05161/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Understanding and Overcoming the Challenges of Efficient Transformer Quantization](https://mlsanity.com/paper/2109.12948/) (2021)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)

## Built on by (most cited first, 20 of 20)

- [On the Impact of Calibration Data in Post-training Quantization and Pruning](https://mlsanity.com/paper/2311.09755/) (2023)
- [Low-Rank Quantization-Aware Training for LLMs](https://mlsanity.com/paper/2406.06385/) (2024)
- [LQER: Low-Rank Quantization Error Reconstruction for LLMs](https://mlsanity.com/paper/2402.02446/) (2024)
- [AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference](https://mlsanity.com/paper/2411.09909/) (2024)
- [Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization](https://mlsanity.com/paper/2503.18599/) (2025)
- [Post Training Quantization of Large Language Models with Microscaling Formats](https://mlsanity.com/paper/2405.07135/) (2024)
- [LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices](https://mlsanity.com/paper/2407.11534/) (2024)
- [QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2502.12346/) (2025)
- [Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment](https://mlsanity.com/paper/2407.03051/) (2024)
- [Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization](https://mlsanity.com/paper/2510.10618/) (2025)
- [RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Model](https://mlsanity.com/paper/2406.15734/) (2024)
- [DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization](https://mlsanity.com/paper/2412.20185/) (2024)
- [A Structure-Aware Framework for Learning Device Placements on Computation Graphs](https://mlsanity.com/paper/2405.14185/) (2024)
- [QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models](https://mlsanity.com/paper/2412.11629/) (2024)
- [Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation](https://mlsanity.com/paper/2510.19498/) (2025)
- [Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2](https://mlsanity.com/paper/2512.23367/) (2025)
- [AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning](https://mlsanity.com/paper/2411.13814/) (2024)
- [Q&C: When Quantization Meets Cache in Efficient Image Generation](https://mlsanity.com/paper/2503.02508/) (2025)
- [Efficient Reasoning on the Edge](https://mlsanity.com/paper/2603.16867/) (2026)
- [P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration](https://mlsanity.com/paper/2609.34867/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantization (PTQ) in LLMs, specifically 4-bit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency -- a topic less explored compared to weight-only quantization. We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and aligning calibration sequence lengths to target tasks. Moreover, we introduce dINT, a hybrid data format combining integer and denormal representations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evaluations of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arithmetic units compatible with dINT, we further confirm that our methods yield a 2× hardware efficiency improvement compared to 8-bit integer MAC unit.