# Quality issues in Machine Learning Software Systems

arXiv: https://arxiv.org/abs/2208.08982 · HTML: https://mlsanity.com/paper/2208.08982/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 85 references that are in the map.
- First posted on arXiv 18 Aug 2022.

## Builds on (most cited first, 22 of 22)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [Neural Network Attributions: A Causal Perspective](https://mlsanity.com/paper/1902.02302/) (2019)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology](https://mlsanity.com/paper/2003.05155/) (2020)
- [Technology Readiness Levels for AI & ML](https://mlsanity.com/paper/2006.12497/) (2020)
- [Studying Software Engineering Patterns for Designing Machine Learning Systems](https://mlsanity.com/paper/1910.04736/) (2019)
- [Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study](https://mlsanity.com/paper/2103.09783/) (2021)
- [The Prevalence of Code Smells in Machine Learning projects](https://mlsanity.com/paper/2103.04146/) (2021)
- [Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inference](https://mlsanity.com/paper/2006.04730/) (2020)
- [Design Smells in Deep Learning Programs: An Empirical Study](https://mlsanity.com/paper/2107.02279/) (2021)
- [Clones in Deep Learning Code: What, Where, and Why?](https://mlsanity.com/paper/2107.13614/) (2021)
- [Dataset Definition Standard (DDS)](https://mlsanity.com/paper/2101.03020/) (2021)

## Built on by (most cited first, 7 of 7)

- [METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities](https://mlsanity.com/paper/2312.06056/) (2023)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Model Callers for Transforming Predictive and Generative AI Applications](https://mlsanity.com/paper/2406.15377/) (2024)
- [Is PMBOK Guide the Right Fit for AI? Re-evaluating Project Management in the Face of Artificial Intelligence Projects](https://mlsanity.com/paper/2506.02214/) (2025)
- [Data-aware Static Analysis: Improving Detection of Semantic Faults in Machine Learning Code Using Data Characteristics](https://mlsanity.com/paper/2606.09957/) (2026)
- [From Failure to Alignment: A Requirements Engineering Framework for Machine Learning Systems](https://mlsanity.com/paper/2606.31589/) (2026)
- [From AI Technical Debt to Agentic Technical Debt: A Systematic Mapping of Root Causes and Manifestations in Agentic AI Systems](https://mlsanity.com/paper/2608.01001/) (2026)

## Abstract (arXiv)

Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are implemented as software components and deployed in Machine Learning Software Systems (MLSSs). Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life. The quality assurance of MLSSs is considered as a challenging task and currently is a hot research topic. Moreover, it is important to cover all various aspects of the quality in MLSSs. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of bad-practices related to poor quality in MLSSs. Method: We plan to conduct a set of interviews with practitioners/experts, believing that interviews are the best method to retrieve their experience and practices when dealing with quality issues. We expect that the catalog of issues developed at this step will also help us later to identify the severity, root causes, and possible remedy for quality issues of MLSSs, allowing us to develop efficient quality assurance tools for ML models and MLSSs.