mlsanityData to 14 Oct 2026

Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective

arXiv 2508.16712 ↗First posted 22 Aug 2025cs.PF cs.AI cs.ARcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their heavy resource demands make quantization-reducing precision to lower-bit formats-critical for efficient serving. While many quantization methods exist, a systematic understanding of their performance, energy, and quality tradeoffs in realistic serving conditions remains a gap. In this work, we first develop a fully automated online characterization framework qMeter, and then conduct an in-depth characterization of 11 post-training LLM quantization methods across 4 model sizes (7B-70B) and two GPU architectures (A100, H100). We evaluate quantization at the application, workload, parallelism, and hardware levels under online serving conditions. Our study reveals highly task- and method-dependent tradeoffs, strong sensitivity to workload characteristics, and complex interactions with parallelism and GPU architecture. We further present three optimization case studies illustrating deployment challenges in capacity planning, energy-efficient scheduling, and multi-objective tuning. To the best of our knowledge, this is one of the first comprehensive application-, system-, and hardware-level characterization of LLM quantization from a joint performance, energy, and quality perspective.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 40 of its 70 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

40 of 70 references.

References · 40

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
28 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  2. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  8. 2021A White Paper on Neural Network Quantizationcited by 472
  9. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  11. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  12. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  13. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163
  14. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  15. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  16. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  17. 2024Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completioncited by 40
  18. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
  19. 2024Characterizing LLM Abstention Behavior in Science QA with Context Perturbationscited by 26
  20. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 14
  21. 2024LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Servicescited by 14
  22. 2024Towards Sustainable Large Language Model Servingcited by 12
  23. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  24. 2025Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit Viewcited by 11
  25. 2023Characterizing Large Language Models as Rationalizers of Knowledge-intensive Taskscited by 9
  26. 2025Brevity is the soul of sustainability: Characterizing LLM response lengthscited by 9
  27. 2024An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2cited by 4
  28. 2025Not All Water Consumption Is Equal: A Water Stress Weighted Metric for Sustainable Computingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefillcited by 4
  2. 2025Toward Cybersecurity-Expert Small Language Modelscited by 4
  3. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  4. 2026Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computerscited by 1
  5. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0
  6. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  7. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designscited by 0
  8. 2026Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisionscited by 0
  9. 2026Analysis of Quantized and Efficiently Adapted Protein Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  2. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  3. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  4. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  5. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
  6. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  7. 2025NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs
  8. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization