mlsanityData to 14 Oct 2026

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

arXiv 2407.10960 ↗First posted 15 Jul 2024cs.LG cs.CL cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

The deployment of large language models (LLMs) is often constrained by memory bandwidth, where the primary bottleneck is the cost of transferring model parameters from the GPU's global memory to its registers. When coupled with custom kernels that fuse the dequantization and matmul operations, weight-only quantization can thus enable faster inference by reducing the amount of memory movement. However, developing high-performance kernels for weight-quantized LLMs presents substantial challenges, especially when the weights are compressed to non-evenly-divisible bit widths (e.g., 3 bits) with non-uniform, lookup table (LUT) quantization. This paper describes FLUTE, a flexible lookup table engine for LUT-quantized LLMs, which uses offline restructuring of the quantized weight matrix to minimize bit manipulations associated with unpacking, and vectorization and duplication of the lookup table to mitigate shared memory bandwidth constraints. At batch sizes < 32 and quantization group size of 128 (typical in LLM inference), the FLUTE kernel can be 2-4x faster than existing GEMM kernels. As an application of FLUTE, we explore a simple extension to lookup table-based NormalFloat quantization and apply it to quantize LLaMA3 to various configurations, obtaining competitive quantization performance against strong baselines while obtaining an end-to-end throughput increase of 1.5 to 2 times.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

22 of 37 references.

References · 22

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  8. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  9. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  10. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  11. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  12. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
10 more
  1. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  2. 2019Quantization Networkscited by 153
  3. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  4. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 99
  5. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  6. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 42
  7. 2023Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPUcited by 23
  8. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23
  9. 2024FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designcited by 12
  10. 2019Iteratively Training Look-Up Tables for Network Quantizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 12.

29 papers.

Sort:

By area:

  1. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  2. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  3. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  4. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 24
  5. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  6. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?cited by 9
  7. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 8
  8. 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
  9. 2026OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformerscited by 6
  10. 2024GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inferencecited by 5
  11. 2024An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2cited by 4
  12. 2025Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizercited by 4
17 more
  1. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  2. 2024An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networkscited by 3
  3. 2025GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Modelscited by 3
  4. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  5. 2025NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMscited by 1
  6. 2025Optimizing Neural Networks with Learnable Non-Linear Activation Functions via Lookup-Based FPGA Accelerationcited by 1
  7. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decodingcited by 1
  8. 20261-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantizationcited by 1
  9. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  10. 2026A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inferencecited by 1
  11. 2026Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weightscited by 1
  12. 2026Fast NF4 Dequantization Kernels for Large Language Model Inferencecited by 0
  13. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  14. 2026UltraQuant: 4-bit KV Caching for Context-Heavy Agentscited by 0
  15. 2026CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weightscited by 0
  16. 2026FlashQuant: Sparse-Dense Fusion for Memory-Efficient Outlier-Aware LLM Inferencecited by 0
  17. 2026SSR: Sparse Segment Reduction for Ternary GEMM Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
  2. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  3. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference
  4. 2025Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices
  5. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  6. 2026Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference
  7. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
  8. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference