# A Human-Centric Assessment Framework for AI

arXiv: https://arxiv.org/abs/2205.12749 · HTML: https://mlsanity.com/paper/2205.12749/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 6 of 6)

- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [Does Explainable Artificial Intelligence Improve Human Decision-Making?](https://mlsanity.com/paper/2006.11194/) (2020)
- [A Turing Test for Transparency](https://mlsanity.com/paper/2106.11394/) (2021)

## Built on by (most cited first, 2 of 2)

- [Bridging the Transparency Gap: What Can Explainable AI Learn From the AI Act?](https://mlsanity.com/paper/2302.10766/) (2023)
- [Linking Surface Facts to Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2310.14909/) (2023)

## Abstract (arXiv)

With the rise of AI systems in real-world applications comes the need for reliable and trustworthy AI. An essential aspect of this are explainable AI systems. However, there is no agreed standard on how explainable AI systems should be assessed. Inspired by the Turing test, we introduce a human-centric assessment framework where a leading domain expert accepts or rejects the solutions of an AI system and another domain expert. By comparing the acceptance rates of provided solutions, we can assess how the AI system performs compared to the domain expert, and whether the AI system's explanations (if provided) are human-understandable. This setup -- comparable to the Turing test -- can serve as a framework for a wide range of human-centric AI system assessments. We demonstrate this by presenting two instantiations: (1) an assessment that measures the classification accuracy of a system with the option to incorporate label uncertainties; (2) an assessment where the usefulness of provided explanations is determined in a human-centric manner.