mlsanityData to 14 Oct 2026

Saying, Not Knowing: Aggressively GGUF-Quantized Small Language Models Still Write Rare Words They Can No Longer Define

arXiv 2610.04403 ↗First posted 3 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Post-training quantization to the GGUF format's mixed-precision K-quants is commonly how open-weight language models reach consumer hardware, yet its effect on fine-grained lexical competence is uncharacterized. We audit 27 quantized artifacts across 13 families and four architecture backbones, 0.35B-14B parameters, evaluated down their published ladder to Q2_K (about 2.6 bits per weight), on 429 frequency-validated rare English words under two probes: surface inclusion of a prompt-supplied word and its one-sentence definition, scored by a tiered multi-synonym matcher, its error measured by a blind LLM-judge census of every definition, with human verification. Three regimes emerge at Q2: total collapse into unusable builds, severe semantic dissociation in sub-2B models, and mostly robust preservation above about 3B. In every sub-2B artifact, definitions fall 20-67% below the artifact's baseline, typically several times the inclusion loss. Two controls separate rarity from task difficulty: within the rare set, loss rises with rarity in six of seven sub-2B artifacts, and on a 100-word common-word set rare words lose more than common words in all eight, significantly in six. Tokenizer vocabulary size does not predict the damage (Spearman rho=0.12); parameter count dominates (rho=0.72), confirmed within five of six same-tokenizer families. Q4_K_M remains lexically clean at >=1B. The damage is frequency-graded, provider-dependent, and not calibrated by WikiText-2 perplexity: across nine artifact-matched ladders, near-identical Q2 penalties (44.7%/47.6%) separate an artifact keeping its definitions (3.6%) from one losing them (43.6%). Aggressively quantized small models can keep generating fluent text while no longer knowing what it means, risking hardware-constrained deployments in domains where semantics carries consequences. Validation must be per artifact.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

32 of 54 references.

References · 32

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2018Model Cards for Model Reportingcited by 1,080
  10. 2022Toy Models of Superpositioncited by 971
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
20 more
  1. 2019What Do Compressed Deep Neural Networks Forget?cited by 171
  2. 2020Characterising Bias in Compressed Modelscited by 167
  3. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  4. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  5. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  6. 2024Evaluating Quantized Large Language Modelscited by 77
  7. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  8. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  9. 2024Accuracy is Not All You Needcited by 31
  10. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 28
  11. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 8
  12. 2026From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantizationcited by 5
  13. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 5
  14. 2025Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Modelscited by 4
  15. 2025Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recallcited by 3
  16. 2026Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Themcited by 2
  17. 2025A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resourcescited by 1
  18. 2025The Uneven Impact of Post-Training Quantization in Machine Translationcited by 1
  19. 2026Towards Automated Lexicography: Generating and Evaluating Definitions for Learner's Dictionariescited by 1
  20. 2026The Asymmetric Harms of LLM Compressioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant
  2. 2026The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
  3. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study
  4. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
  5. 2026Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages
  6. 2026Reliability Scaling Laws for Quantized Large Language Models
  7. 2026The Multilingual Quantization Tax: Structural Collapse and Typological Fragility in Edge SLMs
  8. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models