Quality Issues in Machine Learning Software Systems
Not on the map. Why?
Abstract (arXiv)
Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are implemented as software components and deployed in Machine Learning Software Systems (MLSSs). Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threats to human life. The quality assurance of MLSSs is considered a challenging task and currently is a hot research topic. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of quality issues in MLSSs. Method: We conduct a set of interviews with practitioners/experts, to gather insights about their experience and practices when dealing with quality issues. We validate the identified quality issues via a survey with ML practitioners. Results: Based on the content of 37 interviews, we identified 18 recurring quality issues and 21 strategies to mitigate them. For each identified issue, we describe the causes and consequences according to the practitioners' experience. Conclusion: We believe the catalog of issues developed in this study will allow the community to develop efficient quality assurance tools for ML models and MLSSs. A replication package of our study is available on our public GitHub repository
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 85 references that are in the map. First posted on arXiv 26 Jun 2023.
Builds on
26 of 85 references.
References · 26
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
- 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
- 2019Explainable Machine Learning in Deploymentcited by 210
- 2022Can Foundation Models Wrangle Your Data?cited by 173
- 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 151
- 2021Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspectivecited by 116
- 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
- 2019Neural Network Attributions: A Causal Perspectivecited by 77
- 2021Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Processcited by 56
- 2019Requirements Engineering for Machine Learning: Perspectives from Data Scientistscited by 49
14 more
- 2018On Testing Machine Learning Programscited by 48
- 2020Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodologycited by 41
- 2020Technology Readiness Levels for AI & MLcited by 30
- 2022Operationalizing Machine Learning: An Interview Studycited by 25
- 2019Studying Software Engineering Patterns for Designing Machine Learning Systemscited by 24
- 2021Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Studycited by 21
- 2021The Prevalence of Code Smells in Machine Learning projectscited by 16
- 2023The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practicecited by 14
- 2020Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inferencecited by 11
- 2021Design Smells in Deep Learning Programs: An Empirical Studycited by 10
- 2023Tailoring Requirements Engineering for Responsible AIcited by 7
- 2021Clones in Deep Learning Code: What, Where, and Why?cited by 5
- 2021Dataset Definition Standard (DDS)cited by 3
- 2022The Impact of Personality on Requirements Engineering Activities: A Mixed-Methods Studycited by 1
Built on by
Not cited yet.