mlsanityData to 14 Oct 2026

Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant

arXiv 2409.11055 ↗First posted 17 Sep 2024cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Quantization has gained attention as a promising solution for the cost-effective deployment of large and small language models. However, most prior work has been limited to perplexity or basic knowledge tasks and lacks a comprehensive evaluation of recent models like Llama-3.3. In this paper, we conduct a comprehensive evaluation of instruction-tuned models spanning 1B to 405B parameters, applying four quantization methods across 13 datasets. Our findings reveal that (1) quantized models generally surpass smaller FP16 baselines, yet they often struggle with instruction-following and hallucination detection; (2) FP8 consistently emerges as the most robust option across tasks, and AWQ tends to outperform GPTQ in weight-only quantization; (3) smaller models can suffer severe accuracy drops at 4-bit quantization, while 70B-scale models maintain stable performance; (4) notably, textithard tasks do not always experience the largest accuracy losses, indicating that quantization magnifies a model's inherent weaknesses rather than simply correlating with task difficulty; and (5) an LLM-based judge (MT-Bench) highlights significant performance declines in Coding and STEM tasks, though it occasionally reports improvements in reasoning.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

33 of 40 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
21 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2023A Survey on Model Compression for Large Language Modelscited by 269
  6. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  7. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  8. 2022FP8 Formats for Deep Learningcited by 196
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  11. 2024Evaluating Quantized Large Language Modelscited by 77
  12. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  13. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  14. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 48
  15. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  16. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  17. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  18. 2024Accuracy is Not All You Needcited by 31
  19. 2023LLMEval: A Preliminary Study on How to Evaluate Large Language Modelscited by 23
  20. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11
  21. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspectivecited by 9
  2. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 8
  3. 2026Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deploymentcited by 3
  4. 2026Float8@2bits: Entropy Coding Enables Data-Free Model Compressioncited by 2
  5. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  6. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 1
  7. 2026Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deploymentcited by 1
  8. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0
  9. 2026A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Networkcited by 0
  10. 2026K-Quantization and its Impact on Output Performancecited by 0
  11. 2026Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decodingcited by 0
  12. 2026Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMscited by 0
2 more
  1. 2026Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inferencecited by 0
  2. 2026Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and Englishcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Does quantization affect models' performance on long-context tasks?
  2. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspective
  3. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
  4. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
  5. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  6. 2026Reliability Scaling Laws for Quantized Large Language Models
  7. 2025SiLQ: Simple Large Language Model Quantization-Aware Training
  8. 2024Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models