mlsanityData to 14 Oct 2026

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct

arXiv 2601.14277 ↗First posted 11 Jan 2026cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Quantization is a practical technique for making large language models easier to deploy by reducing the precision used to store and operate on model weights. This can lower memory use and improve runtime feasibility on constrained hardware, which is especially relevant for users running models locally. Quantization in llama.cpp enables large language models to run on commodity hardware, but available formats are often evaluated inconsistently, making it hard to choose among schemes. We present a unified empirical study of the llama.cpp quantization on a single modern model, Llama-3.1-8B-Instruct (FP16, GGUF), covering 3-8 bit K-quant and legacy formats. We evaluate downstream task performance across standard reasoning, knowledge, instruction-following, and truthfulness benchmarks, and also measure perplexity and CPU throughput (prefill/decoding) alongside model size, compression, and quantization time. Ultimately, this work is a practical guide for choosing a llama.cpp quantization scheme, helping readers make informed, context-aware decisions for their intended use and resource budget.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 11 Jan 2026.

Builds on

26 of 29 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
14 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  6. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  7. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  8. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
  9. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  10. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 48
  11. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  12. 2025An Empirical Study of Qwen3 Quantizationcited by 21
  13. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 14
  14. 2025Quantitative Analysis of Performance Drop in DeepSeek Model Quantizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Themcited by 2
  2. 2026NAVI-Orbital: First In-Orbit Demonstration of a Zero-Shot Vision-Language Model for Autonomous Earth Observationcited by 0
  3. 2026Permutation-Based Stegomalware in Large Language Models: Threats and Countermeasurescited by 0
  4. 2026ARID: A Deployable Edge AI System for Structured Information Extraction from Industrial Maintenance Work Orderscited by 0
  5. 2026Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Modelcited by 0
  6. 2026Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Modelscited by 0
  7. 2026Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bitcited by 0
  8. 2026Saying, Not Knowing: Aggressively GGUF-Quantized Small Language Models Still Write Rare Words They Can No Longer Definecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  2. 2025Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models
  3. 2024The Uniqueness of LLaMA3-70B Series with Per-Channel Quantization
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  5. 2025Optimizing LLMs Using Quantization for Mobile Execution
  6. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  7. 2026Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
  8. 2025Does quantization affect models' performance on long-context tasks?