mlsanityData to 7 Oct 2026

LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

arXiv 2206.09557 ↗First posted 20 Jun 2022cs.DC cs.CLcited by 98

Not on the map. Why?

Abstract (arXiv)

Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models introduces a memory wall problem during the generation phase. To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting in practical speed-ups during inference on a single GPU. However, these improvements primarily stem from reduced memory movement, which necessitates a resource-intensive dequantization process rather than actual computational reduction. In this paper, we introduce LUT-GEMM, an efficient kernel for quantized matrix multiplication, which not only eliminates the resource-intensive dequantization process but also reduces computational costs compared to previous kernels for weight-only quantization. Furthermore, we proposed group-wise quantization to offer a flexible trade-off between compression ratio and accuracy. The impact of LUT-GEMM is facilitated by implementing high compression ratios through low-bit quantization and efficient LUT-based operations. We show experimentally that when applied to the OPT-175B model with 3-bit quantization, LUT-GEMM substantially accelerates token generation latency, achieving a remarkable 2.1× improvement on a single GPU when compared to OPTQ, which relies on the costly dequantization process.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 42 of its 70 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

42 of 70 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
30 more
  1. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  6. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  9. 2019The State of Sparsity in Deep Neural Networkscited by 471
  10. 2018Model compression via distillation and quantizationcited by 287
  11. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  13. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  14. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  15. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  16. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  17. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 77
  18. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  19. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  20. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 57
  21. 2016Towards the Limit of Network Quantizationcited by 49
  22. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  23. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  24. 2018Alternating Multi-bit Quantization for Recurrent Neural Networkscited by 30
  25. 2021Kronecker Decomposition for GPT Compressioncited by 27
  26. 2017Network Sketching: Exploiting Binary Structure in Deep CNNscited by 22
  27. 2020BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNscited by 17
  28. 2020Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translationcited by 13
  29. 2018Training wide residual networks for deployment using a single bit for each weightcited by 10
  30. 2017Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 35; 2025: 30; 2026: 15.

98 papers.

Sort:

By area:

  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  8. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  9. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  10. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  11. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
38 more
  1. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  2. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  3. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  4. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  5. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  6. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  7. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 32
  8. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 28
  9. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  10. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
  11. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  12. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  13. 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptationcited by 20
  14. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantizationcited by 20
  15. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  16. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  17. 2025Towards AI Search Paradigmcited by 19
  18. 2024Quantifying the Capabilities of LLMs across Scale and Precisioncited by 17
  19. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 17
  20. 2024SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Modelscited by 16
  21. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  22. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  23. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 14
  24. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  25. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 13
  26. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  27. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matricescited by 11
  28. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  29. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  30. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 10
  31. 20241-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUscited by 9
  32. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Modelscited by 9
  33. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  34. 2025LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Acceleratorcited by 8
  35. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  36. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  37. 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
  38. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 7

50 most cited of 98 All 98 (JSON)