mlsanityData to 7 Oct 2026

Accuracy is Not All You Need

arXiv 2407.09141 ↗First posted 12 Jul 2024cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

When Large Language Models (LLMs) are compressed using techniques such as quantization, the predominant way to demonstrate the validity of such techniques is by measuring the model's accuracy on various benchmarks.If the accuracies of the baseline model and the compressed model are close, it is assumed that there was negligible degradation in quality.However, even when the accuracy of baseline and compressed model are similar, we observe the phenomenon of flips, wherein answers change from correct to incorrect and vice versa in proportion.We conduct a detailed study of metrics across multiple compression techniques, models and datasets, demonstrating that the behavior of compressed models as visible to end-users is often significantly different from the baseline model, even when accuracy is similar.We further evaluate compressed models qualitatively and quantitatively using MT-Bench and show that compressed models are significantly worse than baseline models in this free-form generative task.Thus, we argue that compression techniques should also be evaluated using distance metrics.We propose two such metrics, KL-Divergence and flips, and show that they are well correlated.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

27 of 42 references.

References · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
15 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  5. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  6. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  7. 2020Characterising Bias in Compressed Modelscited by 162
  8. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  9. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  10. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  11. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54
  12. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 41
  13. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  14. 2020Going Beyond Classification Accuracy Metrics in Model Compressioncited by 15
  15. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 21.

27 papers.

Sort:

By area:

  1. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  2. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  3. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 12
  4. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 5
  5. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levelscited by 5
  6. 2025A Proximal Operator for Inducing 2:4-Sparsitycited by 4
  7. 2025SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weightscited by 4
  8. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  9. 2026The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMscited by 3
  10. 2026Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AIcited by 2
  11. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  12. 2026Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Modelscited by 1
15 more
  1. 2026Total Variation Distance Estimation in Autoregressive Modelscited by 1
  2. 2026Which Decisions Low-Bit Quantization Breaks, and How to Predict Themcited by 1
  3. 2026QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstructioncited by 1
  4. 2026Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMscited by 1
  5. 2026Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decodingcited by 0
  6. 2026Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extendedcited by 0
  7. 2026Reliability Scaling Laws for Quantized Large Language Modelscited by 0
  8. 2026HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Taskscited by 0
  9. 2026Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Servingcited by 0
  10. 2026Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Servingcited by 0
  11. 2026Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Themcited by 0
  12. 2026Permutation-Based Stegomalware in Large Language Models: Threats and Countermeasurescited by 0
  13. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  14. 2026The Undetected Damage of Quantization on Retrieval and How to Fix Itcited by 0
  15. 2026G2PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensationcited by 0