# Verifying Machine Learning Interpretability Requirements through Provenance

arXiv: https://arxiv.org/abs/2604.21599 · HTML: https://mlsanity.com/paper/2604.21599/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Builds on (most cited first, 9 of 9)

- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Machine Learning Pipelines: Provenance, Reproducibility and FAIR Data Principles](https://mlsanity.com/paper/2006.12117/) (2020)
- [Management of Machine Learning Lifecycle Artifacts: A Survey](https://mlsanity.com/paper/2210.11831/) (2022)
- [Requirements Engineering for Machine Learning: A Review and Reflection](https://mlsanity.com/paper/2210.00859/) (2022)
- [Non-Functional Requirements for Machine Learning: An Exploration of System Scope and Interest](https://mlsanity.com/paper/2203.11063/) (2022)
- [Inherently Interpretable Tree Ensemble Learning](https://mlsanity.com/paper/2410.19098/) (2024)
- [yProv4ML: Effortless Provenance Tracking for Machine Learning Systems](https://mlsanity.com/paper/2507.01078/) (2025)

## Built on by (most cited first, 1 of 1)

- [Safeguarding LLM Agents from Misalignment through Provenance Analysis](https://mlsanity.com/paper/2607.01236/) (2026)

## Abstract (arXiv)

Machine Learning (ML) Engineering is a growing field that necessitates an increase in the rigor of ML development. It draws many ideas from software engineering and more specifically, from requirements engineering. Existing literature on ML Engineering defines quality models and Non-Functional Requirements (NFRs) specific to ML, in particular interpretability being one such NFR. However, a major challenge occurs in verifying ML NFRs, including interpretability. Although existing literature defines interpretability in terms of ML, it remains an immeasurable requirement, making it impossible to definitively confirm whether a model meets its interpretability requirement. This paper shows how ML provenance can be used to verify ML interpretability requirements. This work provides an approach for how ML engineers can save various types of model and data provenance to make the model's behavior transparent and interpretable. Saving this data forms the basis of quantifiable Functional Requirements (FRs) whose verification in turn verifies the interpretability NFR. Ultimately, this paper contributes a method to verify interpretability NFRs for ML models.