# Towards Guidelines for Assessing Qualities of Machine Learning Systems

arXiv: https://arxiv.org/abs/2008.11007 · HTML: https://mlsanity.com/paper/2008.11007/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2020.

## Built on by (most cited first, 16 of 16)

- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study](https://mlsanity.com/paper/2103.09783/) (2021)
- [A Synthesis of Green Architectural Tactics for ML-Enabled Systems](https://mlsanity.com/paper/2312.09610/) (2023)
- [How Mature is Requirements Engineering for AI-based Systems? A Systematic Mapping Study on Practices, Challenges, and Future Research Directions](https://mlsanity.com/paper/2409.07192/) (2024)
- [Towards a Reference Software Architecture for Human-AI Teaming in Smart Manufacturing](https://mlsanity.com/paper/2201.04876/) (2022)
- [What is Software Quality for AI Engineers? Towards a Thinning of the Fog](https://mlsanity.com/paper/2203.12697/) (2022)
- [MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities](https://mlsanity.com/paper/2303.01998/) (2023)
- [Navigating the EU AI Act: A Methodological Approach to Compliance for Safety-critical Products](https://mlsanity.com/paper/2403.16808/) (2024)
- [A Mathematical Model of the Hidden Feedback Loop Effect in Machine Learning Systems](https://mlsanity.com/paper/2405.02726/) (2024)
- [Towards Sustainability Model Cards](https://mlsanity.com/paper/2507.19559/) (2025)
- [Building AI Innovation Labs together with Companies](https://mlsanity.com/paper/2203.08465/) (2022)
- [Using Quality Attribute Scenarios for ML Model Test Case Generation](https://mlsanity.com/paper/2406.08575/) (2024)
- [Data Quality Awareness: A Journey from Traditional Data Management to Data Science Systems](https://mlsanity.com/paper/2411.03007/) (2024)
- [Quality Model for Machine Learning Components](https://mlsanity.com/paper/2602.05043/) (2026)

## Abstract (arXiv)

Nowadays, systems containing components based on machine learning (ML) methods are becoming more widespread. In order to ensure the intended behavior of a software system, there are standards that define necessary quality aspects of the system and its components (such as ISO/IEC 25010). Due to the different nature of ML, we have to adjust quality aspects or add additional ones (such as trustworthiness) and be very precise about which aspect is really relevant for which object of interest (such as completeness of training data), and how to objectively assess adherence to quality requirements. In this article, we present the construction of a quality model (i.e., evaluation objects, quality aspects, and metrics) for an ML system based on an industrial use case. This quality model enables practitioners to specify and assess quality requirements for such kinds of ML systems objectively. In the future, we want to learn how the term quality differs between different types of ML systems and come up with general guidelines for specifying and assessing qualities of ML systems.