# MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities

arXiv: https://arxiv.org/abs/2303.01998 · HTML: https://mlsanity.com/paper/2303.01998/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology](https://mlsanity.com/paper/2003.05155/) (2020)
- [Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges](https://mlsanity.com/paper/1905.04223/) (2019)
- [Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices](https://mlsanity.com/paper/2006.12358/) (2020)
- [Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry](https://mlsanity.com/paper/1812.05389/) (2018)
- [Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects](https://mlsanity.com/paper/2002.03389/) (2020)
- [Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems](https://mlsanity.com/paper/2103.14101/) (2021)
- [Towards Guidelines for Assessing Qualities of Machine Learning Systems](https://mlsanity.com/paper/2008.11007/) (2020)
- [Deepchecks: A Library for Testing and Validating Machine Learning Models and Data](https://mlsanity.com/paper/2203.08491/) (2022)

## Built on by (most cited first, 2 of 2)

- [Law and the Emerging Political Economy of Algorithmic Audits](https://mlsanity.com/paper/2406.11855/) (2024)
- [Using Quality Attribute Scenarios for ML Model Test Case Generation](https://mlsanity.com/paper/2406.08575/) (2024)

## Abstract (arXiv)

Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we propose MLTE (Machine Learning Test and Evaluation, colloquially referred to as "melt"), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. MLTE tooling supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.