# QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models

arXiv: https://arxiv.org/abs/2310.09259 · HTML: https://mlsanity.com/paper/2310.09259/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [RPTQ: Reorder-based Post-training Quantization for Large Language Models](https://mlsanity.com/paper/2304.01089/) (2023)
- [8-Bit Approximations for Parallelism in Deep Learning](https://mlsanity.com/paper/1511.04561/) (2015)
- [A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset](https://mlsanity.com/paper/1806.00358/) (2018)
- [FPTQ: Fine-grained Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2308.15987/) (2023)

## Built on by (most cited first, 30 of 39)

- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs](https://mlsanity.com/paper/2406.01721/) (2024)
- [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals](https://mlsanity.com/paper/2412.14363/) (2024)
- [Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization](https://mlsanity.com/paper/2404.03605/) (2024)
- [Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs](https://mlsanity.com/paper/2508.14896/) (2025)
- [Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models](https://mlsanity.com/paper/2502.15799/) (2025)
- [DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization](https://mlsanity.com/paper/2511.04063/) (2025)
- [WUSH: Near-Optimal Adaptive Transforms for LLM Quantization](https://mlsanity.com/paper/2512.00956/) (2025)
- [ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks](https://mlsanity.com/paper/2312.08583/) (2023)
- [Post Training Quantization of Large Language Models with Microscaling Formats](https://mlsanity.com/paper/2405.07135/) (2024)
- [PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization](https://mlsanity.com/paper/2410.05265/) (2024)
- [MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models](https://mlsanity.com/paper/2508.02343/) (2025)
- [A Speed Odyssey for Deployable Quantization of LLMs](https://mlsanity.com/paper/2311.09550/) (2023)
- [EfQAT: An Efficient Framework for Quantization-Aware Training](https://mlsanity.com/paper/2411.11038/) (2024)
- [Beyond Outliers: A Study of Optimizers Under Quantization](https://mlsanity.com/paper/2509.23500/) (2025)
- [Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models](https://mlsanity.com/paper/2507.07877/) (2025)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring](https://mlsanity.com/paper/2501.13331/) (2025)
- [Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Size](https://mlsanity.com/paper/2503.04704/) (2025)
- [QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition](https://mlsanity.com/paper/2503.19353/) (2025)
- [Trends in AI Supercomputers](https://mlsanity.com/paper/2504.16026/) (2025)
- [BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers](https://mlsanity.com/paper/2507.03117/) (2025)
- [TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization](https://mlsanity.com/paper/2605.19561/) (2026)
- [Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models](https://mlsanity.com/paper/2405.01943/) (2024)
- [Low-Rank Correction for Quantized LLMs](https://mlsanity.com/paper/2412.07902/) (2024)
- [LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs](https://mlsanity.com/paper/2602.17681/) (2026)
- [TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training](https://mlsanity.com/paper/2604.24088/) (2026)
- [CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs](https://mlsanity.com/paper/2405.17233/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. Yet, the vast majority of existing work focuses on weight-only quantization, which can reduce runtime costs in the memory-bound one-token-at-a-time generative setting, but does not address them in compute-bound scenarios, such as batched inference or prompt processing. In this paper, we address the general quantization problem, where both weights and activations should be quantized. We show, for the first time, that the majority of inference computations for large generative models such as LLaMA, OPT, and Falcon can be performed with both weights and activations being cast to 4 bits, in a way that leads to practical speedups, while at the same time maintaining good accuracy. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit, while keeping some outlier weights and activations in higher-precision. The key feature of our scheme is that it is designed with computational efficiency in mind: we provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x relative to FP16 execution. We provide detailed studies for models from the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate inference using quantization plus 2:4 sparsity. Code is available at: https://github.com/IST-DASLab/QUIK.