# Human-in-the-Loop Interpretability Prior

arXiv: https://arxiv.org/abs/1805.11571 · HTML: https://mlsanity.com/paper/1805.11571/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2018.

## Built on by (most cited first, 30 of 36)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [HIVE: Evaluating the Human Interpretability of Visual Explanations](https://mlsanity.com/paper/2112.03184/) (2021)
- [Synthetic Benchmarks for Scientific Research in Explainable Machine Learning](https://mlsanity.com/paper/2106.12543/) (2021)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Self-Interpretable Model with TransformationEquivariant Interpretation](https://mlsanity.com/paper/2111.04927/) (2021)
- [On the Faithfulness of Vision Transformer Explanations](https://mlsanity.com/paper/2404.01415/) (2024)
- [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021)
- [Evaluating the Interpretability of Generative Models by Interactive Reconstruction](https://mlsanity.com/paper/2102.01264/) (2021)
- [Robust Semantic Interpretability: Revisiting Concept Activation Vectors](https://mlsanity.com/paper/2104.02768/) (2021)
- [Keep CALM and Improve Visual Feature Attribution](https://mlsanity.com/paper/2106.07861/) (2021)
- [POETREE: Interpretable Policy Learning with Adaptive Decision Trees](https://mlsanity.com/paper/2203.08057/) (2022)
- [Interactive Dimensionality Reduction for Comparative Analysis](https://mlsanity.com/paper/2106.15481/) (2021)
- [Improving Human Sequential Decision-Making with Reinforcement Learning](https://mlsanity.com/paper/2108.08454/) (2021)
- [On the Interpretability of Part-Prototype Based Classifiers: A Human Centric Analysis](https://mlsanity.com/paper/2310.06966/) (2023)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [Perspectives on Incorporating Expert Feedback into Model Updates](https://mlsanity.com/paper/2205.06905/) (2022)
- [A Review of Explainable Artificial Intelligence in Manufacturing](https://mlsanity.com/paper/2107.02295/) (2021)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Diagrammatization and Abduction to Improve AI Interpretability With Domain-Aligned Explanations for Medical Diagnosis](https://mlsanity.com/paper/2302.01241/) (2023)
- [Unraveling Human-AI Teaming: A Review and Outlook](https://mlsanity.com/paper/2504.05755/) (2025)
- [Improving Prototypical Visual Explanations with Reward Reweighing, Reselection, and Retraining](https://mlsanity.com/paper/2307.03887/) (2023)
- [Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing](https://mlsanity.com/paper/2309.05679/) (2023)
- [A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning](https://mlsanity.com/paper/2406.00116/) (2024)
- [LCE: A Framework for Explainability of DNNs for Ultrasound Image Based on Concept Discovery](https://mlsanity.com/paper/2408.09899/) (2024)
- [Bandits for Learning to Explain from Explanations](https://mlsanity.com/paper/2102.03815/) (2021)
- [Enriching Artificial Intelligence Explanations with Knowledge Fragments](https://mlsanity.com/paper/2204.05579/) (2022)
- [Automatic explanation of the classification of Spanish legal judgments in jurisdiction-dependent law categories with tree estimators](https://mlsanity.com/paper/2404.00437/) (2024)
- [Towards Model-informed Precision Dosing with Expert-in-the-loop Machine Learning](https://mlsanity.com/paper/2106.14384/) (2021)
- [Debiased-CAM to mitigate systematic error with faithful visual explanations of machine learning](https://mlsanity.com/paper/2201.12835/) (2022)

## Abstract (arXiv)

We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.