mlsanityData to 14 Oct 2026

An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face

arXiv 2512.07983 ↗First posted 8 Dec 2025cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

As machine learning (ML) becomes an integral part of high-autonomy systems, it is critical to ensure the trustworthiness of learning-enabled software systems (LESS). Yet, the nondeterministic and run-time-defined semantics of ML complicate traditional software refactoring. We define semantic preservation in LESS as the property that optimizations of intelligent components do not alter the system's overall functional behavior. This paper introduces an empirical framework to evaluate semantic preservation in LESS by mining model evolution data from HuggingFace. We extract commit histories, textitModel Cards, and performance metrics from a large number of models. To establish baselines, we conducted case studies in three domains, tracing performance changes across versions. Our analysis demonstrates how textitsemantic drift can be detected via evaluation metrics across commits and reveals common refactoring patterns based on commit message analysis. Although API constraints limited the possibility of estimating a full-scale threshold, our pipeline offers a foundation for defining community-accepted boundaries for semantic preservation. Our contributions include: (1) a large-scale dataset of ML model evolution, curated from 1.7 million Hugging Face entries via a reproducible pipeline using the native HF hub API, (2) a practical pipeline for the evaluation of semantic preservation for a subset of 536 models and 4000+ metrics and (3) empirical case studies illustrating semantic drift in practice. Together, these contributions advance the foundations for more maintainable and trustworthy ML systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

15 of 26 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018Model Cards for Model Reportingcited by 1,080
  3. 2020Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challengescited by 120
  4. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  5. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  6. 2022Towards Training Reproducible Deep Learning Modelscited by 23
  7. 2023Analyzing the Evolution and Maintenance of ML Models on Hugging Facecited by 22
  8. 2021Decomposing Convolutional Neural Networks into Reusable and Replaceable Modulescited by 14
  9. 2022If a Human Can See It, So Should Your System: Reliability Requirements for Machine Vision Componentscited by 11
  10. 2023The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Productscited by 10
  11. 2023Fix Fairness, Don't Ruin Accuracy: Performance Aware Fairness Repair using AutoMLcited by 9
  12. 2023Empirical Optimal Risk to Quantify Model Trustworthiness for Failure Detectioncited by 4
3 more
  1. 2024Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approachcited by 4
  2. 2022An Empirical Study on the Bugs Found while Reusing Pre-trained Natural Language Processing Modelscited by 2
  3. 2023Detecting out-of-distribution text using topological features of transformer-based language modelscited by 2

Built on by

Not cited yet.