mlsanityData to 14 Oct 2026

Using Quality Attribute Scenarios for ML Model Test Case Generation

arXiv 2406.08575 ↗First posted 12 Jun 2024cs.SE cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Testing of machine learning (ML) models is a known challenge identified by researchers and practitioners alike. Unfortunately, current practice for ML model testing prioritizes testing for model performance, while often neglecting the requirements and constraints of the ML-enabled system that integrates the model. This limited view of testing leads to failures during integration, deployment, and operations, contributing to the difficulties of moving models from development to production. This paper presents an approach based on quality attribute (QA) scenarios to elicit and define system- and model-relevant test cases for ML models. The QA-based approach described in this paper has been integrated into MLTE, a process and tool to support ML model test and evaluation. Feedback from users of MLTE highlights its effectiveness in testing beyond model performance and identifying failures early in the development process.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

14 of 23 references.

References · 14

  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  2. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  3. 2019Explainable Machine Learning in Deploymentcited by 211
  4. 2018On Testing Machine Learning Programscited by 48
  5. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 27
  6. 2023A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitionerscited by 22
  7. 2021Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Developmentcited by 21
  8. 2021Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systemscited by 19
  9. 2020Towards Guidelines for Assessing Qualities of Machine Learning Systemscited by 16
  10. 2024PyTester: Deep Reinforcement Learning for Text-to-Testcase Generationcited by 6
  11. 2022Capabilities for Better ML Engineeringcited by 3
  12. 2023Identifying Concerns When Specifying Machine Learning-Enabled Systems: A Perspective-Based Approachcited by 3
2 more
  1. 2022What is Software Quality for AI Engineers? Towards a Thinning of the Fogcited by 2
  2. 2023MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualitiescited by 2

Built on by

Not cited yet.