# How model accuracy and explanation fidelity influence user trust

arXiv: https://arxiv.org/abs/1907.12652 · HTML: https://mlsanity.com/paper/1907.12652/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2019.

## Built on by (most cited first, 30 of 45)

- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Improving Expert Predictions with Conformal Prediction](https://mlsanity.com/paper/2201.12006/) (2022)
- [The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations](https://mlsanity.com/paper/2205.03295/) (2022)
- [Trust in AI: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2403.14680/) (2024)
- [Human-Aligned Calibration for AI-Assisted Decision Making](https://mlsanity.com/paper/2306.00074/) (2023)
- [Designing Decision Support Systems Using Counterfactual Prediction Sets](https://mlsanity.com/paper/2306.03928/) (2023)
- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [A Diachronic Perspective on User Trust in AI under Uncertainty](https://mlsanity.com/paper/2310.13544/) (2023)
- [The Impact of Imperfect XAI on Human-AI Decision-Making](https://mlsanity.com/paper/2307.13566/) (2023)
- [Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making](https://mlsanity.com/paper/2308.04375/) (2023)
- [Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions](https://mlsanity.com/paper/2107.09734/) (2021)
- [Diagnosing AI Explanation Methods with Folk Concepts of Behavior](https://mlsanity.com/paper/2201.11239/) (2022)
- [What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on QA Systems](https://mlsanity.com/paper/2305.14331/) (2023)
- [The Importance of Human-Labeled Data in the Era of LLMs](https://mlsanity.com/paper/2306.14910/) (2023)
- [Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration](https://mlsanity.com/paper/2409.12809/) (2024)
- [Why do explanations fail? A typology and discussion on failures in XAI](https://mlsanity.com/paper/2405.13474/) (2024)
- [Pitfalls of Explainable ML: An Industry Perspective](https://mlsanity.com/paper/2106.07758/) (2021)
- [Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis](https://mlsanity.com/paper/2309.11987/) (2023)
- [Survey on AI Ethics: A Socio-technical Perspective](https://mlsanity.com/paper/2311.17228/) (2023)
- [Evaluating the Explainability of Attributes and Prototypes for a Medical Classification Model](https://mlsanity.com/paper/2404.09917/) (2024)
- [Evaluative Item-Contrastive Explanations in Rankings](https://mlsanity.com/paper/2312.10094/) (2023)
- [Beware of "Explanations" of AI](https://mlsanity.com/paper/2504.06791/) (2025)
- [Latent Debate: A Surrogate Framework for Interpreting LLM Thinking](https://mlsanity.com/paper/2512.01909/) (2025)
- [Concept-based Explanations using Non-negative Concept Activation Vectors and Decision Tree for CNN Models](https://mlsanity.com/paper/2211.10807/) (2022)
- [Automatic Textual Explanations of Concept Lattices](https://mlsanity.com/paper/2304.08093/) (2023)
- [Conformal Set-based Human-AI Complementarity with Multiple Experts](https://mlsanity.com/paper/2508.06997/) (2025)
- [Contextual Trust](https://mlsanity.com/paper/2303.08900/) (2023)

## Abstract (arXiv)

Machine learning systems have become popular in fields such as marketing, financing, or data mining. While they are highly accurate, complex machine learning systems pose challenges for engineers and users. Their inherent complexity makes it impossible to easily judge their fairness and the correctness of statistically learned relations between variables and classes. Explainable AI aims to solve this challenge by modelling explanations alongside with the classifiers, potentially improving user trust and acceptance. However, users should not be fooled by persuasive, yet untruthful explanations. We therefore conduct a user study in which we investigate the effects of model accuracy and explanation fidelity, i.e. how truthfully the explanation represents the underlying model, on user trust. Our findings show that accuracy is more important for user trust than explainability. Adding an explanation for a classification result can potentially harm trust, e.g. when adding nonsensical explanations. We also found that users cannot be tricked by high-fidelity explanations into having trust for a bad classifier. Furthermore, we found a mismatch between observed (implicit) and self-reported (explicit) trust.