# Deceptive AI Explanations: Creation and Detection

arXiv: https://arxiv.org/abs/2001.07641 · HTML: https://mlsanity.com/paper/2001.07641/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2020.

## Built on by (most cited first, 14 of 14)

- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [Unfooling Perturbation-Based Post Hoc Explainers](https://mlsanity.com/paper/2205.14772/) (2022)
- [When and How to Fool Explainable Models (and Humans) with Adversarial Examples](https://mlsanity.com/paper/2107.01943/) (2021)
- [Explaining Classifiers by Constructing Familiar Concepts](https://mlsanity.com/paper/2203.04109/) (2022)
- [Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy](https://mlsanity.com/paper/2402.07632/) (2024)
- [Believe The HiPe: Hierarchical Perturbation for Fast, Robust, and Model-Agnostic Saliency Mapping](https://mlsanity.com/paper/2103.05108/) (2021)
- [Concept-based Adversarial Attacks: Tricking Humans and Classifiers Alike](https://mlsanity.com/paper/2203.10166/) (2022)
- [Foundation models in brief: A historical, socio-technical focus](https://mlsanity.com/paper/2212.08967/) (2022)
- [Deceptive Automated Interpretability: Language Models Coordinating to Fool Oversight Systems](https://mlsanity.com/paper/2504.07831/) (2025)
- [Don't Lie to Me: Avoiding Malicious Explanations with STEALTH](https://mlsanity.com/paper/2301.10407/) (2023)
- [VitrAI -- Applying Explainable AI in the Real World](https://mlsanity.com/paper/2102.06518/) (2021)
- [Implementing Smart Contracts: The case of NFT-rental with pay-per-like](https://mlsanity.com/paper/2308.02424/) (2023)

## Abstract (arXiv)

Artificial intelligence (AI) comes with great opportunities but can also pose significant risks. Automatically generated explanations for decisions can increase transparency and foster trust, especially for systems based on automated predictions by AI models. However, given, e.g., economic incentives to create dishonest AI, to what extent can we trust explanations? To address this issue, our work investigates how AI models (i.e., deep learning, and existing instruments to increase transparency regarding AI decisions) can be used to create and detect deceptive explanations. As an empirical evaluation, we focus on text classification and alter the explanations generated by GradCAM, a well-established explanation technique in neural networks. Then, we evaluate the effect of deceptive explanations on users in an experiment with 200 participants. Our findings confirm that deceptive explanations can indeed fool humans. However, one can deploy machine learning (ML) methods to detect seemingly minor deception attempts with accuracy exceeding 80% given sufficient domain knowledge. Without domain knowledge, one can still infer inconsistencies in the explanations in an unsupervised manner, given basic knowledge of the predictive model under scrutiny.