# What do we know about Hugging Face? A systematic literature review and quantitative validation of qualitative claims arXiv: https://arxiv.org/abs/2406.08205 ยท HTML: https://mlsanity.com/paper/2406.08205/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 69 references that are in the map. - First posted on arXiv 12 Jun 2024. ## Builds on (most cited first, 23 of 23) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019) - [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021) - [An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry](https://mlsanity.com/paper/2303.02552/) (2023) - [Small World with High Risks: A Study of Security Threats in the npm Ecosystem](https://mlsanity.com/paper/1902.09217/) (2019) - [Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on Hugging Face](https://mlsanity.com/paper/2401.13822/) (2024) - [Analyzing the Evolution and Maintenance of ML Models on Hugging Face](https://mlsanity.com/paper/2311.13380/) (2023) - [An Empirical Comparison of Dependency Network Evolution in Seven Software Packaging Ecosystems](https://mlsanity.com/paper/1710.04936/) (2017) - [Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends](https://mlsanity.com/paper/2401.13177/) (2024) - [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023) - [Reusing Deep Learning Models: Challenges and Directions in Software Engineering](https://mlsanity.com/paper/2404.16688/) (2024) - [PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software](https://mlsanity.com/paper/2402.00699/) (2024) - [Demystifying the Vulnerability Propagation and Its Evolution via Dependency Trees in the NPM Ecosystem](https://mlsanity.com/paper/2201.03981/) (2022) - [GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration](https://mlsanity.com/paper/2306.01481/) (2023) - [PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages](https://mlsanity.com/paper/2303.08934/) (2023) - ["I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency](https://mlsanity.com/paper/2310.01642/) (2023) - [Lessons Learned from Mining the Hugging Face Repository](https://mlsanity.com/paper/2402.07323/) (2024) - [Paving the Way for Mature Secondary Research: The Seven Types of Literature Review](https://mlsanity.com/paper/2205.01163/) (2022) - [On the evolution of technical lag in the npm package dependency network](https://mlsanity.com/paper/1806.01545/) (2018) - [An Empirical Analysis of the Python Package Index (PyPI)](https://mlsanity.com/paper/1907.11073/) (2019) - [On the Diversity of Software Package Popularity Metrics: An Empirical Study of npm](https://mlsanity.com/paper/1901.04217/) (2019) - [Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability](https://mlsanity.com/paper/2303.02551/) (2023) - [What are the characteristics of highly-selected packages? A case study on the npm ecosystem](https://mlsanity.com/paper/2204.04562/) (2022) ## Built on by (most cited first, 9 of 9) - [An Empirical Analysis of Machine Learning Model and Dataset Documentation, Supply Chain, and Licensing Challenges on Hugging Face](https://mlsanity.com/paper/2502.04484/) (2025) - [Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2505.19532/) (2025) - [Economies of Open Intelligence: Tracing Power & Participation in the Model Ecosystem](https://mlsanity.com/paper/2512.03073/) (2025) - [Compliance Cards: Automated EU AI Act Compliance Analyses amidst a Complex AI Supply Chain](https://mlsanity.com/paper/2406.14758/) (2024) - [How do Machine Learning Models Change?](https://mlsanity.com/paper/2411.09645/) (2024) - [Consiglieres in the Shadow: Understanding the Use of Uncensored Large Language Models in Cybercrimes](https://mlsanity.com/paper/2508.12622/) (2025) - [On the (In)Security of Loading Machine Learning Models](https://mlsanity.com/paper/2509.06703/) (2025) - [Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?](https://mlsanity.com/paper/2512.23385/) (2025) - [Computational Compliance for AI Regulation: Blueprint for a New Research Domain](https://mlsanity.com/paper/2601.04474/) (2026) ## Abstract (arXiv) Background: Collaborative Software Package Registries (SPRs) are an integral part of the software supply chain. Much engineering work synthesizes SPR package into applications. Prior research has examined SPRs for traditional software, such as NPM (JavaScript) and PyPI (Python). Pre-Trained Model (PTM) Registries are an emerging class of SPR of increasing importance, because they support the deep learning supply chain. Aims: Recent empirical research has examined PTM registries in ways such as vulnerabilities, reuse processes, and evolution. However, no existing research synthesizes them to provide a systematic understanding of the current knowledge. Some of the existing research includes qualitative claims lacking quantitative analysis. Our research fills these gaps by providing a knowledge synthesis and quantitative analyses. Methods: We first conduct a systematic literature review (SLR). We then observe that some of the claims are qualitative. We identify quantifiable metrics associated with those claims, and measure in order to substantiate these claims. Results: From our SLR, we identify 12 claims about PTM reuse on the HuggingFace platform, 4 of which lack quantitative validation. We successfully test 3 of these claims through a quantitative analysis, and directly compare one with traditional software. Our findings corroborate qualitative claims with quantitative measurements. Our findings are: (1) PTMs have a much higher turnover rate than traditional software, indicating a dynamic and rapidly evolving reuse environment within the PTM ecosystem; and (2) There is a strong correlation between documentation quality and PTM popularity. Conclusions: We confirm qualitative research claims with concrete metrics, supporting prior qualitative and case study research. Our measures show further dynamics of PTM reuse, inspiring research infrastructure and new measures.