mlsanityData to 14 Oct 2026

Explanation User Interfaces: A Systematic Literature Review

arXiv 2505.20085 ↗First posted 26 May 2025cs.HC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Artificial Intelligence (AI) is one of the major technological advancements of this century, bearing incredible potential for users through AI-powered applications and tools in numerous domains. Being often black-box (i.e., its decision-making process is unintelligible), developers typically resort to eXplainable Artificial Intelligence (XAI) techniques to interpret the behaviour of AI models to produce systems that are transparent, fair, reliable, and trustworthy. However, presenting explanations to the user is not trivial and is often left as a secondary aspect of the system's design process, leading to AI systems that are not useful to end-users. This paper presents a Systematic Literature Review on Explanation User Interfaces (XUIs) to gain a deeper understanding of the solutions and design guidelines employed in the academic literature to effectively present explanations to users. To improve the contribution and real-world impact of this survey, we also present a platform to support Human-cEnteRed developMent of Explainable user interfaceS (HERMES) and guide practitioners and scholars in the design and evaluation of XUIs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 197 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

49 of 197 references.

References · 49

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  4. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  5. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  7. 2016The Mythos of Model Interpretabilitycited by 721
  8. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 614
  9. 2016Rationalizing Neural Predictionscited by 281
  10. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  11. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  12. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
37 more
  1. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  2. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  3. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 85
  4. 2018RuleMatrix: Visualizing and Understanding Classifiers with Rulescited by 62
  5. 2020CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualizationcited by 57
  6. 2020DECE: Decision Explorer with Counterfactual Explanations for Machine Learning Modelscited by 38
  7. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
  8. 2021M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysiscited by 29
  9. 2024From Explainable to Interactive AI: A Literature Review on Current Trends in Human-AI Interactioncited by 26
  10. 2019Visual Interaction with Deep Learning Models through Collaborative Semantic Inferencecited by 24
  11. 2020The Grammar of Interactive Explanatory Model Analysiscited by 24
  12. 2017A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanationscited by 20
  13. 2022Explainable Artificial Intelligence (XAI) from a user perspective- A synthesis of prior literature and problematizing avenues for future researchcited by 20
  14. 2021Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Makingcited by 19
  15. 2022ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspectivecited by 18
  16. 2023Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorationscited by 18
  17. 2020Why an Android App is Classified as Malware? Towards Malware Classification Interpretationcited by 16
  18. 2021Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learningcited by 16
  19. 2024EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurationscited by 14
  20. 2021VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Modelscited by 12
  21. 2024User Characteristics in Explainable AI: The Rabbit Hole of Personalization?cited by 12
  22. 2021From Philosophy to Interfaces: an Explanatory Method and a Tool Inspired by Achinstein's Theory of Explanationcited by 8
  23. 2024Exploiting Preference Elicitation in Interactive and User-centered Algorithmic Recourse: An Initial Explorationcited by 8
  24. 2023DeepSeer: Interactive RNN Explanation and Debugging via State Abstractioncited by 6
  25. 2021Effects of interactivity and presentation on review-based explanations for recommendationscited by 5
  26. 2022Visual Exploration of Machine Learning Model Behavior with Hierarchical Surrogate Rule Setscited by 5
  27. 2023Interactive Explanation with Varying Level of Details in an Explainable Scientific Literature Recommender Systemcited by 5
  28. 2025Explainable AI-Based Interface System for Weather Forecasting Modelcited by 4
  29. 2020LIMEADE: From AI Explanations to Advice Takingcited by 3
  30. 2020A Visual Analytics Framework for Contrastive Network Analysiscited by 3
  31. 2023Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labelingcited by 3
  32. 2023TimeTuner: Diagnosing Time Representations for Time-Series Forecasting with Counterfactual Explanationscited by 3
  33. 2024What is the focus of XAI in UI design? Prioritizing UI design principles for enhancing XAI user experiencecited by 3
  34. 2021Explainability via Interactivity? Supporting Nonexperts' Sensemaking of Pretrained CNN by Interacting with Their Daily Surroundingscited by 2
  35. 2022Generating and Visualizing Trace Link Explanationscited by 2
  36. 2023VISHIEN-MAAT: Scrollytelling visualization design for explaining Siamese Neural Network concept to non-technical userscited by 1
  37. 2023TRIVEA: Transparent Ranking Interpretation using Visual Explanation of Black-Box Algorithmic Rankerscited by 1

Built on by

Not cited yet.