# Using Quality Attribute Scenarios for ML Model Test Case Generation

arXiv: https://arxiv.org/abs/2406.08575 · HTML: https://mlsanity.com/paper/2406.08575/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development](https://mlsanity.com/paper/2110.02932/) (2021)
- [Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems](https://mlsanity.com/paper/2103.14101/) (2021)
- [Towards Guidelines for Assessing Qualities of Machine Learning Systems](https://mlsanity.com/paper/2008.11007/) (2020)
- [PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation](https://mlsanity.com/paper/2401.07576/) (2024)
- [Capabilities for Better ML Engineering](https://mlsanity.com/paper/2211.06409/) (2022)
- [Identifying Concerns When Specifying Machine Learning-Enabled Systems: A Perspective-Based Approach](https://mlsanity.com/paper/2309.07980/) (2023)
- [What is Software Quality for AI Engineers? Towards a Thinning of the Fog](https://mlsanity.com/paper/2203.12697/) (2022)
- [MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities](https://mlsanity.com/paper/2303.01998/) (2023)

## Abstract (arXiv)

Testing of machine learning (ML) models is a known challenge identified by researchers and practitioners alike. Unfortunately, current practice for ML model testing prioritizes testing for model performance, while often neglecting the requirements and constraints of the ML-enabled system that integrates the model. This limited view of testing leads to failures during integration, deployment, and operations, contributing to the difficulties of moving models from development to production. This paper presents an approach based on quality attribute (QA) scenarios to elicit and define system- and model-relevant test cases for ML models. The QA-based approach described in this paper has been integrated into MLTE, a process and tool to support ML model test and evaluation. Feedback from users of MLTE highlights its effectiveness in testing beyond model performance and identifying failures early in the development process.