mlsanityData to 7 Oct 2026

GPTVQ: The Blessing of Dimensionality for LLM Quantization

arXiv 2402.15319 ↗First posted 23 Feb 2024cs.LG cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose the GPTVQ method, a new fast method for post-training vector quantization (VQ) that scales well to Large Language Models (LLMs). Our method interleaves quantization of one or more columns with updates to the remaining unquantized weights, using information from the Hessian of the per-layer output reconstruction MSE. Quantization codebooks are initialized using an efficient data-aware version of the EM algorithm. The codebooks are then updated, and further compressed by using integer quantization and SVD-based compression. GPTVQ establishes a new state-of-the art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2 and Mistral. Furthermore, our method is efficient: on a single H100 it takes between 3 and 11 hours to process a Llamav2-70B model, depending on quantization setting. Lastly, with on-device timings for VQ decompression on a mobile CPU we show that VQ leads to improved latency compared to using a 4-bit integer format.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

40 of 57 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Mistral 7Bcited by 3,065
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
28 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2024Mixtral of Expertscited by 1,785
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2021A White Paper on Neural Network Quantizationcited by 453
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  13. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  14. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  15. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  16. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  17. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  18. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  19. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  20. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  21. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  22. 2024Towards Conversational Diagnostic AIcited by 79
  23. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  24. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  25. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  26. 2020Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networkscited by 28
  27. 2023LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookupcited by 19
  28. 2023Bias in Pruned Vision Models: In-Depth Analysis and Countermeasurescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 32; 2026: 33.

78 papers.

Sort:

By area:

  1. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  2. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  3. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  4. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  5. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 53
  6. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  7. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  8. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidancecited by 32
  9. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 28
  10. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  11. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  12. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 20
38 more
  1. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  2. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  3. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
  4. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 9
  5. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  6. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  7. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Modelscited by 7
  8. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6
  9. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 6
  10. 2026Leech Lattice Vector Quantization for Efficient LLM Compressioncited by 6
  11. 2025PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decouplingcited by 5
  12. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  13. 2026CoreQ: Learning-Free Mismatch Correction and Successive Rounding for Quantizationcited by 5
  14. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  15. 2024Pyramid Vector Quantization for LLMscited by 4
  16. 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
  17. 2025Radio: Rate-Distortion Optimization for Large Language Model Compressioncited by 4
  18. 2025RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantizationcited by 4
  19. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  20. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compressioncited by 4
  21. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  22. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 4
  23. 2026Statistically-Lossless Quantization of Large Language Modelscited by 4
  24. 2026HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantizationcited by 4
  25. 2025ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mambacited by 3
  26. 2025CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMscited by 3
  27. 2025CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compressioncited by 3
  28. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMscited by 3
  29. 2026LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotationcited by 3
  30. 2024TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstructioncited by 2
  31. 2024Low-Rank Correction for Quantized LLMscited by 2
  32. 2025DiscQuant: A Quantization Method for Neural Networks Inspired by Discrepancy Theorycited by 2
  33. 2025Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learningcited by 2
  34. 2025RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Modelscited by 2
  35. 2025MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Networkcited by 2
  36. 2026Saliency-Aware Regularized Quantization Calibration for Large Language Modelscited by 2
  37. 2026EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecturecited by 2
  38. 2026LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantizationcited by 2

50 most cited of 78 All 78 (JSON)