mlsanityData to 14 Oct 2026

"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

arXiv 2411.02355 ↗First posted 4 Nov 2024cs.LG cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear. In this paper, we conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks on the entire Llama-3.1 model family. Through over 500,000 evaluations, our investigation yields several key findings: (1) FP8 (W8A8-FP) is effectively lossless across all model scales, (2) well-tuned INT8 (W8A8-INT) achieves surprisingly low (1-3%) accuracy degradation, and (3) INT4 weight-only (W4A16-INT) is more competitive than expected, rivaling 8-bit quantization. Further, we investigate the optimal quantization format for different deployments by analyzing inference performance through the popular vLLM framework. Our analysis provides clear deployment recommendations: W4A16 is the most cost-efficient for synchronous setups, while W8A8 dominates in asynchronous continuous batching. For mixed workloads, the optimal choice depends on the specific use case. Our findings offer practical, data-driven guidelines for deploying quantized LLMs at scale -- ensuring the best balance between speed, efficiency, and accuracy.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

20 of 29 references.

References · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
8 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  3. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  6. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  7. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  8. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 25.

34 papers.

Sort:

By area:

  1. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  2. 2025The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithmcited by 20
  3. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 15
  4. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  5. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  6. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 10
  7. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 8
  8. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 7
  9. 2026Statistically-Lossless Quantization of Large Language Modelscited by 6
  10. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  11. 2025OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literaturecited by 4
  12. 2025Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backendscited by 4
22 more
  1. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  2. 2026The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMscited by 4
  3. 2026MatGPTQ: Efficient and Accurate Inference over Nested Quantized Modelscited by 3
  4. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 3
  5. 2026Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deploymentcited by 3
  6. 2025Designing Synthetic Discussion Generation Systems: A Case Study for Online Facilitationcited by 2
  7. 2026Float8@2bits: Entropy Coding Enables Data-Free Model Compressioncited by 2
  8. 2026FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inferencecited by 2
  9. 2026QuantClaw: Precision Where It Matters for OpenClawcited by 1
  10. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 1
  11. 2026IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifiercited by 1
  12. 2026Optimizing AI Inference Across the Deployment Stackcited by 1
  13. 2025CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMscited by 0
  14. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0
  15. 2026LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillationcited by 0
  16. 2026K-Quantization and its Impact on Output Performancecited by 0
  17. 2026Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Servingcited by 0
  18. 2026Attention Quantization for Tabular Foundation Modelscited by 0
  19. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  20. 2026A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarkscited by 0
  21. 2026Bits Under ZK-LLM: Evaluating Zero-Knowledge-Friendly Quantization for Verifiable Private LLM Inferencecited by 0
  22. 2026How Divergence Becomes Decision Flips in Compressed Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  2. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  3. 2023Efficient LLM Inference on CPUs
  4. 2025Optimizing Large Language Model Training Using FP4 Quantization
  5. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  6. 2024Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs
  7. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
  8. 2023Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?