# A Large-Scale Study of Model Integration in ML-Enabled Software Systems

arXiv: https://arxiv.org/abs/2408.06226 · HTML: https://mlsanity.com/paper/2408.06226/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 98 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence](https://mlsanity.com/paper/2002.04803/) (2020)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Vision-and-Language Pretrained Models: A Survey](https://mlsanity.com/paper/2204.07356/) (2022)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry](https://mlsanity.com/paper/2303.02552/) (2023)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [Engineering AI Systems: A Research Agenda](https://mlsanity.com/paper/2001.07522/) (2020)
- [ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detection](https://mlsanity.com/paper/2106.08890/) (2021)
- [What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform](https://mlsanity.com/paper/1811.07643/) (2018)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Sampling Projects in GitHub for MSR Studies](https://mlsanity.com/paper/2103.04682/) (2021)
- [Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development](https://mlsanity.com/paper/2110.02932/) (2021)
- [The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large](https://mlsanity.com/paper/2112.01590/) (2021)
- [Are Self-Driving Cars Secure? Evasion Attacks against Deep Neural Networks for Steering Angle Prediction](https://mlsanity.com/paper/1904.07370/) (2019)
- [A systematic literature review on source code similarity measurement and clone detection: techniques, applications, and challenges](https://mlsanity.com/paper/2306.16171/) (2023)
- [Do Code Clones Matter?](https://mlsanity.com/paper/1701.05472/) (2017)
- [The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Products](https://mlsanity.com/paper/2308.04328/) (2023)
- [A large-scale comparative analysis of Coding Standard conformance in Open-Source Data Science projects](https://mlsanity.com/paper/2007.08978/) (2020)
- [PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software](https://mlsanity.com/paper/2402.00699/) (2024)
- [Asset Management in Machine Learning: A Survey](https://mlsanity.com/paper/2102.06919/) (2021)
- [Feature Interactions on Steroids: On the Composition of ML Models](https://mlsanity.com/paper/2105.06449/) (2021)
- [Studying the Characteristics of AIOps Projects on GitHub](https://mlsanity.com/paper/2212.13245/) (2022)
- [An Empirical Study of Challenges in Machine Learning Asset Management](https://mlsanity.com/paper/2402.15990/) (2024)
- [Empirical Analysis on CI/CD Pipeline Evolution in Machine Learning Projects](https://mlsanity.com/paper/2403.12199/) (2024)
- [ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study](https://mlsanity.com/paper/2401.06513/) (2024)

## Built on by (most cited first, 6 of 6)

- [Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2411.19463/) (2024)
- [How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration](https://mlsanity.com/paper/2502.00429/) (2025)
- [LLMs in Mobile Apps: Practices, Challenges, and Opportunities](https://mlsanity.com/paper/2502.15908/) (2025)
- [Architecturally Significant MLOps Guidelines for ML Model Integration and Deployment: a Gray Literature Review](https://mlsanity.com/paper/2606.06535/) (2026)
- [Beyond Models: Reflections on Engineering AI-enabled Systems in a Project-Based Course](https://mlsanity.com/paper/2606.16842/) (2026)
- [Software Engineering for AI-driven Building Operation](https://mlsanity.com/paper/2608.16237/) (2026)

## Abstract (arXiv)

The rise of machine learning (ML) and its integration into software systems has drastically changed development practices. While software engineering traditionally focused on manually created code artifacts with dedicated processes and architectures, ML-enabled systems require additional data-science methods and tools to create ML artifacts -- especially ML models and training data. However, integrating models into systems, and managing the many different artifacts involved, is far from trivial. ML-enabled systems can easily have multiple ML models that interact with each other and with traditional code in intricate ways. Unfortunately, while challenges and practices of building ML-enabled systems have been studied, little is known about the characteristics of real-world ML-enabled systems beyond isolated examples. Improving engineering processes and architectures for ML-enabled systems requires improving the empirical understanding of these systems. We present a large-scale study of 2,928 open-source ML-enabled software systems. We classified and analyzed them to determine system characteristics, model and code reuse practices, and architectural aspects of integrating ML models. Our findings show that these systems still mainly consist of traditional source code, and that ML model reuse through code duplication or pre-trained models is common. We also identified different ML integration patterns and related implementation practices. We hope that our results help improve practices for integrating ML models, bringing data science and software engineering closer together.