# Explainability Pitfalls: Beyond Dark Patterns in Explainable AI

arXiv: https://arxiv.org/abs/2109.12480 · HTML: https://mlsanity.com/paper/2109.12480/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 76 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach](https://mlsanity.com/paper/2002.01092/) (2020)
- [Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study](https://mlsanity.com/paper/2002.00772/) (2020)
- [Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions](https://mlsanity.com/paper/1901.03729/) (2019)
- [The Pragmatic Turn in Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2002.09595/) (2020)
- [The age of secrecy and unfairness in recidivism prediction](https://mlsanity.com/paper/1811.00731/) (2018)
- [How to Support Users in Understanding Intelligent Systems? Structuring the Discussion](https://mlsanity.com/paper/2001.08301/) (2020)
- [Dark Patterns, Electronic Medical Records, and the Opioid Epidemic](https://mlsanity.com/paper/2105.08870/) (2021)

## Built on by (most cited first, 30 of 36)

- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI](https://mlsanity.com/paper/2302.00799/) (2023)
- [Selective Explanations: Leveraging Human Input to Align Explainable AI](https://mlsanity.com/paper/2301.09656/) (2023)
- [Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2209.11812/) (2022)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Seamful XAI: Operationalizing Seamful Design in Explainable AI](https://mlsanity.com/paper/2211.06753/) (2022)
- ["There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making](https://mlsanity.com/paper/2205.05758/) (2022)
- [Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation](https://mlsanity.com/paper/2402.01705/) (2024)
- [The Conflict Between Explainable and Accountable Decision-Making Algorithms](https://mlsanity.com/paper/2205.05306/) (2022)
- [Explainability in reinforcement learning: perspective and position](https://mlsanity.com/paper/2203.11547/) (2022)
- [Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration](https://mlsanity.com/paper/2409.12809/) (2024)
- [An Ontology of Co-Creative AI Systems](https://mlsanity.com/paper/2310.07472/) (2023)
- [Explainable artificial intelligence approaches for brain-computer interfaces: a review and design space](https://mlsanity.com/paper/2312.13033/) (2023)
- [On the Relationship Between Explanations, Fairness Perceptions, and Decisions](https://mlsanity.com/paper/2204.13156/) (2022)
- [User-centric evaluation of explainability of AI with and for humans: a comprehensive empirical study](https://mlsanity.com/paper/2410.15952/) (2024)
- [AI Alignment Breaks at the Edge](https://mlsanity.com/paper/2602.20042/) (2026)
- [Towards Balancing Preference and Performance through Adaptive Personalized Explainability](https://mlsanity.com/paper/2504.13856/) (2025)
- [The Case Against Explainability](https://mlsanity.com/paper/2305.12167/) (2023)
- [The European Commitment to Human-Centered Technology: The Integral Role of HCI in the EU AI Act's Success](https://mlsanity.com/paper/2402.14728/) (2024)
- [People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI](https://mlsanity.com/paper/2403.08828/) (2024)
- [Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models](https://mlsanity.com/paper/2404.07725/) (2024)
- [Not All Explanations are Created Equal: Investigating the Pitfalls of Current XAI Evaluation](https://mlsanity.com/paper/2511.03730/) (2025)
- [A Human-Centric Perspective on Fairness and Transparency in Algorithmic Decision-Making](https://mlsanity.com/paper/2205.00033/) (2022)
- [Local Universal Explainer (LUX) -- a rule-based explainer with factual, counterfactual and visual explanations](https://mlsanity.com/paper/2310.14894/) (2023)
- [What Is the Point of Equality in Machine Learning Fairness? Beyond Equality of Opportunity](https://mlsanity.com/paper/2506.16782/) (2025)
- [Beyond Satisfaction: From Placebic to Actionable Explanations For Enhanced Understandability](https://mlsanity.com/paper/2512.06591/) (2025)
- [Beyond Following: Mixing Active Initiative into Computational Creativity](https://mlsanity.com/paper/2409.16291/) (2024)

## Abstract (arXiv)

To make Explainable AI (XAI) systems trustworthy, understanding harmful effects is just as important as producing well-designed explanations. In this paper, we address an important yet unarticulated type of negative effect in XAI. We introduce explainability pitfalls(EPs), unanticipated negative downstream effects from AI explanations manifesting even when there is no intention to manipulate users. EPs are different from, yet related to, dark patterns, which are intentionally deceptive practices. We articulate the concept of EPs by demarcating it from dark patterns and highlighting the challenges arising from uncertainties around pitfalls. We situate and operationalize the concept using a case study that showcases how, despite best intentions, unsuspecting negative effects such as unwarranted trust in numerical explanations can emerge. We propose proactive and preventative strategies to address EPs at three interconnected levels: research, design, and organizational.