# Quality Issues in Machine Learning Software Systems

arXiv: https://arxiv.org/abs/2306.15007 · HTML: https://mlsanity.com/paper/2306.15007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 85 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 26 of 26)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [Neural Network Attributions: A Causal Perspective](https://mlsanity.com/paper/1902.02302/) (2019)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology](https://mlsanity.com/paper/2003.05155/) (2020)
- [Technology Readiness Levels for AI & ML](https://mlsanity.com/paper/2006.12497/) (2020)
- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [Studying Software Engineering Patterns for Designing Machine Learning Systems](https://mlsanity.com/paper/1910.04736/) (2019)
- [Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study](https://mlsanity.com/paper/2103.09783/) (2021)
- [The Prevalence of Code Smells in Machine Learning projects](https://mlsanity.com/paper/2103.04146/) (2021)
- [The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice](https://mlsanity.com/paper/2301.09001/) (2023)
- [Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inference](https://mlsanity.com/paper/2006.04730/) (2020)
- [Design Smells in Deep Learning Programs: An Empirical Study](https://mlsanity.com/paper/2107.02279/) (2021)
- [Tailoring Requirements Engineering for Responsible AI](https://mlsanity.com/paper/2302.10816/) (2023)
- [Clones in Deep Learning Code: What, Where, and Why?](https://mlsanity.com/paper/2107.13614/) (2021)
- [Dataset Definition Standard (DDS)](https://mlsanity.com/paper/2101.03020/) (2021)
- [The Impact of Personality on Requirements Engineering Activities: A Mixed-Methods Study](https://mlsanity.com/paper/2210.07807/) (2022)

## Abstract (arXiv)

Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are implemented as software components and deployed in Machine Learning Software Systems (MLSSs). Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threats to human life. The quality assurance of MLSSs is considered a challenging task and currently is a hot research topic. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of quality issues in MLSSs. Method: We conduct a set of interviews with practitioners/experts, to gather insights about their experience and practices when dealing with quality issues. We validate the identified quality issues via a survey with ML practitioners. Results: Based on the content of 37 interviews, we identified 18 recurring quality issues and 21 strategies to mitigate them. For each identified issue, we describe the causes and consequences according to the practitioners' experience. Conclusion: We believe the catalog of issues developed in this study will allow the community to develop efficient quality assurance tools for ML models and MLSSs. A replication package of our study is available on our public GitHub repository