mlsanityData to 14 Oct 2026

CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs

arXiv 2512.17970 ↗First posted 19 Dec 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Weight-only quantization is widely used to mitigate the memory-bound nature of LLM inference. Codebook-based methods extend this trend by achieving strong accuracy in the extremely low-bit regime (e.g., 2-bit). However, current kernels rely on dequantization, which repeatedly fetches centroids and reconstructs weights, incurring substantial latency and cache pressure. We present CodeGEMM, a codebook-centric GEMM kernel that replaces dequantization with precomputed inner products between centroids and activations stored in a lightweight Psumbook. At inference, code indices directly gather these partial sums, eliminating per-element lookups and reducing the on-chip footprint. The kernel supports the systematic exploration of latency-memory-accuracy trade-offs under a unified implementation. On Llama-3 models, CodeGEMM delivers 1.83x (8B) and 8.93x (70B) speedups in the 2-bit configuration compared to state-of-the-art codebook-based quantization at comparable accuracy and further improves computing efficiency and memory subsystem utilization.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

24 of 30 references.

References · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2024Massive Activations in Large Language Modelscited by 294
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
12 more
  1. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  2. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  3. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  4. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  5. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  6. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  7. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
  8. 2023FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantizationcited by 48
  9. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 30
  10. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  11. 2025FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tablescited by 9
  12. 2023Look-Up mAI GeMM: Increasing AI GeMMs Performance by Nearly 2.5x via msGeMMcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecturecited by 2
  2. 2026A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inferencecited by 1
  3. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
  2. 2026PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
  3. 2026TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs
  4. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference
  5. 2025MVDRAM: Enabling GeMV Execution in Unmodified DRAM for Low-Bit LLM Acceleration
  6. 2023A Speed Odyssey for Deployable Quantization of LLMs
  7. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
  8. 2025OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration