mlsanityData to 7 Oct 2026

MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities

arXiv 2303.01998 ↗First posted 3 Mar 2023cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we propose MLTE (Machine Learning Test and Evaluation, colloquially referred to as "melt"), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. MLTE tooling supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 47 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

19 of 47 references.

References · 19

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2018Model Cards for Model Reportingcited by 1,068
  5. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  6. 2021Unsolved Problems in ML Safetycited by 291
  7. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  8. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 151
  9. 2021Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Processcited by 56
  10. 2019Requirements Engineering for Machine Learning: Perspectives from Data Scientistscited by 49
  11. 2018On Testing Machine Learning Programscited by 48
  12. 2020Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodologycited by 41
7 more
  1. 2019Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challengescited by 39
  2. 2018Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industrycited by 38
  3. 2020Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practicescited by 38
  4. 2020Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projectscited by 24
  5. 2021Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systemscited by 19
  6. 2020Towards Guidelines for Assessing Qualities of Machine Learning Systemscited by 16
  7. 2022Deepchecks: A Library for Testing and Validating Machine Learning Models and Datacited by 7

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Law and the Emerging Political Economy of Algorithmic Auditscited by 5
  2. 2024Using Quality Attribute Scenarios for ML Model Test Case Generationcited by 0