# OpenHEXAI: An Open-Source Framework for Human-Centered Evaluation of Explainable Machine Learning

arXiv: https://arxiv.org/abs/2403.05565 · HTML: https://mlsanity.com/paper/2403.05565/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 30 of 30)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- ['It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions](https://mlsanity.com/paper/1801.10408/) (2018)
- [Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study](https://mlsanity.com/paper/2002.00772/) (2020)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making](https://mlsanity.com/paper/2101.05303/) (2021)
- [Do Explanations make VQA Models more Predictable to a Human?](https://mlsanity.com/paper/1810.12366/) (2018)
- [Why Does My Model Fail? Contrastive Local Explanations for Retail Forecasting](https://mlsanity.com/paper/1908.00085/) (2019)
- [Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards](https://mlsanity.com/paper/2108.07374/) (2021)
- [Assessing the Local Interpretability of Machine Learning Models](https://mlsanity.com/paper/1902.03501/) (2019)
- [A Human-Grounded Evaluation of SHAP for Alert Processing](https://mlsanity.com/paper/1907.03324/) (2019)
- [Use-Case-Grounded Simulations for Explanation Evaluation](https://mlsanity.com/paper/2206.02256/) (2022)
- [ExSum: From Local Explanations to Model Understanding](https://mlsanity.com/paper/2205.00130/) (2022)
- [A Psychological Theory of Explainability](https://mlsanity.com/paper/2205.08452/) (2022)
- [Progressive Disclosure: Designing for Effective Transparency](https://mlsanity.com/paper/1811.02164/) (2018)

## Built on by (most cited first, 2 of 2)

- [Data Science Principles for Interpretable and Explainable AI](https://mlsanity.com/paper/2405.10552/) (2024)
- [Confident Teacher, Confident Student? A Novel User Study Design for Investigating the Didactic Potential of Explanations and their Impact on Uncertainty](https://mlsanity.com/paper/2409.17157/) (2024)

## Abstract (arXiv)

Recently, there has been a surge of explainable AI (XAI) methods driven by the need for understanding machine learning model behaviors in high-stakes scenarios. However, properly evaluating the effectiveness of the XAI methods inevitably requires the involvement of human subjects, and conducting human-centered benchmarks is challenging in a number of ways: designing and implementing user studies is complex; numerous design choices in the design space of user study lead to problems of reproducibility; and running user studies can be challenging and even daunting for machine learning researchers. To address these challenges, this paper presents OpenHEXAI, an open-source framework for human-centered evaluation of XAI methods. OpenHEXAI features (1) a collection of diverse benchmark datasets, pre-trained models, and post hoc explanation methods; (2) an easy-to-use web application for user study; (3) comprehensive evaluation metrics for the effectiveness of post hoc explanation methods in the context of human-AI decision making tasks; (4) best practice recommendations of experiment documentation; and (5) convenient tools for power analysis and cost estimation. OpenHEAXI is the first large-scale infrastructural effort to facilitate human-centered benchmarks of XAI methods. It simplifies the design and implementation of user studies for XAI methods, thus allowing researchers and practitioners to focus on the scientific questions. Additionally, it enhances reproducibility through standardized designs. Based on OpenHEXAI, we further conduct a systematic benchmark of four state-of-the-art post hoc explanation methods and compare their impacts on human-AI decision making tasks in terms of accuracy, fairness, as well as users' trust and understanding of the machine learning model.