Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them
Not on the map. Why?
Abstract (arXiv)
Developers increasingly run large language models locally by pulling quantized GGUF artifacts from public registries, yet nothing in the distribution pipeline functionally tests these conversions before they reach users. We executed 327 quantized code-capable model artifacts: 305 from the official Ollama library, spanning 15 model lines at every eligible quantization level at or under 8 GB, and 22 from the most-downloaded community repositories on HuggingFace. Each ran a 15-task smoke suite calibrated so that healthy artifacts pass while a known-broken one fails; suspects then faced full 164-task evaluation, a second inference backend, an independent distributor's conversion of the same model and quantization as referee, and, for community files, re-testing under the artifact's own template. The official library carries five silently defective artifacts, a batch of four Qwen2.5-Coder-3B conversions and one phi3.5-mini conversion, that solve zero of 164 tasks and zero of the smoke suite on both backends while independent conversions of the same models work: 1.6% of official artifacts, 2 of 29 model-and-size conversion groups. The adjudication chain cleared small-model artifacts that a naive threshold would condemn as broken when they are merely collapsed by extreme quantization, and it exposed two older community conversions that degrade badly on CUDA yet pass on Metal: not defective files but backend-dependent failures, a third phenomenon no registry currently tests for. Two confirmed defects produce output whose surface statistics sit inside the healthy range, invisible to any low-noise heuristic short of execution. We release the audit dataset, the quantcheck acceptance-testing tool, and disclosure reports for every confirmed defect (https://github.com/aditi-p31/quantcheck), and argue that model registries need the acceptance gate that package registries already run.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 5 Sep 2026.
Builds on
28 of 37 references.
References · 28
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
- 2024Evaluating Quantized Large Language Modelscited by 77
- 2024Exploiting LLM Quantizationcited by 65
- 2022Taxonomy of Attacks on Open-Source Software Supply Chainscited by 58
- 2023An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registrycited by 47
- 2024How Does Quantization Affect Multilingual LLMs?cited by 43
16 more
- 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 34
- 2024Accuracy is Not All You Needcited by 31
- 2024Large Language Model Supply Chain: A Research Agendacited by 23
- 2023Analyzing the Evolution and Maintenance of ML Models on Hugging Facecited by 22
- 2023An Empirical Study of Malicious Code In PyPI Ecosystemcited by 19
- 2024Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubscited by 18
- 2021What are Weak Links in the npm Supply Chain?cited by 16
- 2024A Large-Scale Exploit Instrumentation Study of AI/ML Supply Chain Attacks in Hugging Face Modelscited by 14
- 2024Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trendscited by 12
- 2022An Empirical Study of Challenges in Converting Deep Learning Modelscited by 8
- 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 8
- 2023"I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistencycited by 7
- 2024A Survey on Data Quality Dimensions and Tools for Machine Learningcited by 7
- 2020Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defectscited by 5
- 2025Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Studycited by 4
- 2023Fault Localization for Buggy Deep Learning Framework Conversions in Image Recognitioncited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs
- 2026The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
- 2026Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
- 2026Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
- 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
- 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
- 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs
- 2026TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance