Explainable Deep Learning: A Field Guide for the Uninitiated
Not on the map. Why?
Abstract (arXiv)
Deep neural networks (DNNs) have become a proven and indispensable machine learning tool. As a black-box model, it remains difficult to diagnose what aspects of the model's input drive the decisions of a DNN. In countless real-world domains, from legislation and law enforcement to healthcare, such diagnosis is essential to ensure that DNN decisions are driven by aspects appropriate in the context of its use. The development of methods and studies enabling the explanation of a DNN's decisions has thus blossomed into an active, broad area of research. A practitioner wanting to study explainable deep learning may be intimidated by the plethora of orthogonal directions the field has taken. This complexity is further exacerbated by competing definitions of what it means ``to explain'' the actions of a DNN and to evaluate an approach's ``ability to explain''. This article offers a field guide to explore the space of explainable deep learning aimed at those uninitiated in the field. The field guide: i) Introduces three simple dimensions defining the space of foundational methods that contribute to explainable deep learning, ii) discusses the evaluations for model explanations, iii) places explainability in the context of other related deep learning research areas, and iv) finally elaborates on user-oriented explanation designing and potential future directions on explainable deep learning. We hope the guide is used as an easy-to-digest starting point for those just embarking on research in this field.
Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 27; 2022: 32; 2023: 28; 2024: 25; 2025: 10; 2026: 5.
127 papers.
Sort:
By area:
- 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
- 2022Teaching language models to support answers with verified quotescited by 290
- 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
- 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
- 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
- 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
- 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
- 2022Logic-Based Explainability in Machine Learningcited by 47
- 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
- 2022On Tackling Explanation Redundancy in Decision Treescited by 44
- 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
38 more
- 2021Tell me why! Explanations support learning relational and causal structurecited by 36
- 2022A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challengescited by 30
- 2023Artificial intelligence to advance Earth observation: : A review of models, recent trends, and pathways forwardcited by 28
- 2021Interpretable agent communication from scratch (with a generic visual processor emerging on the side)cited by 25
- 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
- 2022Fairness and Explainability: Bridging the Gap Towards Fair Model Explanationscited by 22
- 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
- 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 18
- 2022Astronomia ex machina: a history, primer, and outlook on neural networks in astronomycited by 17
- 2023Learning by Self-Explainingcited by 17
- 2021SurvNAM: The machine learning survival model explanationcited by 16
- 2021XAI Handbook: Towards a Unified Framework for Explainable AIcited by 14
- 2022Survey on Fair Reinforcement Learning: Theory and Practicecited by 14
- 2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networkscited by 13
- 2021Explainable Artificial Intelligence for Human Decision-Support System in Medical Domaincited by 12
- 2022What You See is What You Classify: Black Box Attributionscited by 12
- 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
- 2022Explainability in reinforcement learning: perspective and positioncited by 11
- 2022Greybox XAI: a Neural-Symbolic learning framework to produce interpretable predictions for image classificationcited by 11
- 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Frameworkcited by 11
- 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
- 2021When and How to Fool Explainable Models (and Humans) with Adversarial Examplescited by 10
- 2022Limitations of Deep Learning for Inverse Problems on Digital Hardwarecited by 9
- 2022Constrained Parameter Inference as a Principle for Learningcited by 9
- 2021Ensembles of Random SHAPscited by 8
- 2021Nested Multiple Instance Learning with Attention Mechanismscited by 7
- 2024Efficient Deep Learning with Decorrelated Backpropagationcited by 7
- 2024ECG Unveiled: Analysis of Client Re-identification Risks in Real-World ECG Datasetscited by 7
- 2021An Imprecise SHAP as a Tool for Explaining the Class Probability Distributions under Limited Training Datacited by 6
- 2022Gradient Backpropagation based Feature Attribution to Enable Explainable-AI on the Edgecited by 6
- 2023Improving Interpretability via Explicit Word Interaction Graph Layercited by 6
- 2023SAR-UNet: Small Attention Residual UNet for Explainable Nowcasting Taskscited by 6
- 2023Interpretable Mesomorphic Networks for Tabular Datacited by 6
- 2021Exploring Explainable AI in the Financial Sector: Perspectives of Banks and Supervisory Authoritiescited by 5
- 2022Explainable Misinformation Detection Across Multiple Social Media Platformscited by 5
- 2023FederatedTrust: A Solution for Trustworthy Federated Learningcited by 5
- 2023Non-Uniform Interpolation in Integrated Gradients for Low-Latency Explainable-AIcited by 5
- 2023Explainable AI applications in the Medical Domain: a systematic reviewcited by 5
50 most cited of 127 All 127 (JSON)
- 2026Explaining Reinforcement Learning Agents via Inductive Logic Programming15 Jul 2026
- 2026Learning Quantifiable Visual Explanations Without Ground-Truth18 May 2026
- 2026How Can One Choose the Best CAM-Based Explainability Method for a CNN Model?3 May 2026
- 2026Trustworthy Data-driven Chronological Age Estimation from Panoramic Dental Images19 Jan 2026
- 2026Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie18 Jan 2026
- 2025A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts17 Sep 2025
- 2025Neuromorphic Intelligence15 Sep 2025
- 2025DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification6 Aug 2025
- 2025A Survey on Interpretability in Visual Recognition15 Jul 2025
- 2025Gradient Attention Map Based Verification of Deep Convolutional Neural Networks with Application to X-ray Image Datasets29 Apr 2025
- 2025Readable Twins of Unreadable Models17 Apr 2025
- 2025HyConEx: Hypernetwork classifier with counterfactual explanations for tabular data16 Mar 2025
38 more
- 2025Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey12 Mar 2025
- 2025Complex Networks for Pattern-Based Data Classification25 Feb 2025
- 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis3 Feb 2025
- 2024FastRM: An efficient and automatic explainability framework for multimodal generative models2 Dec 2024
- 2024The Return of Pseudosciences in Artificial Intelligence: Have Machine Learning and Deep Learning Forgotten Lessons from Statistics and History?27 Nov 2024
- 2024Clinicians' Voice: Fundamental Considerations for XAI in Healthcare7 Nov 2024
- 2024Explainability in AI Based Applications: A Framework for Comparing Different Techniques28 Oct 2024
- 2024Interpretable end-to-end Neurosymbolic Reinforcement Learning agents18 Oct 2024
- 2024Improving Prototypical Parts Abstraction for Case-Based Reasoning Explanations Designed for the Kidney Stone Type Recognition19 Sep 2024
- 2024Gradient-free Post-hoc Explainability Using Distillation Aided Learnable Approach17 Sep 2024
- 2024Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantization12 Aug 2024
- 2024ECG Unveiled: Analysis of Client Re-identification Risks in Real-World ECG Datasets2 Aug 2024
- 2024XAI-Guided Enhancement of Vegetation Indices for Crop Mapping11 Jul 2024
- 2024Explainable Image Recognition via Enhanced Slot-attention Based Classifier8 Jul 2024
- 2024DocXplain: A Novel Model-Agnostic Explainability Method for Document Image Classification4 Jul 2024
- 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning6 Jun 2024
- 2024Discovering Continuous-Time Memory-Based Symbolic Policies using Genetic Programming4 Jun 2024
- 2024Fairness-Optimized Synthetic EHR Generation for Arbitrary Downstream Predictive Tasks4 Jun 2024
- 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning23 May 2024
- 2024Efficient Deep Learning with Decorrelated Backpropagation3 May 2024
- 2024Towards a Research Community in Interpretable Reinforcement Learning: the InterpPol Workshop16 Apr 2024
- 2024Hindsight PRIORs for Reward Learning from Human Preferences12 Apr 2024
- 2024Leveraging Counterfactual Paths for Contrastive Explanations of POMDP Policies28 Mar 2024
- 2024Causal hybrid modeling with double machine learning20 Feb 2024
- 2024Dual feature-based and example-based explanation methods29 Jan 2024
- 2024Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement18 Jan 2024
- 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents11 Jan 2024
- 2024α-HMM: A Graphical Model for RNA Folding7 Jan 2024
- 2023On the Relationship Between Interpretability and Explainability in Machine Learning20 Nov 2023
- 2023One-dimensional convolutional neural network model for breast cancer subtypes classification and biochemical content evaluation using micro-FTIR hyperspectral images23 Oct 2023
- 2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks18 Oct 2023
- 2023ProtoExplorer: Interpretable Forensic Analysis of Deepfake Videos using Prototype Exploration and Refinement20 Sep 2023
- 2023Learning by Self-Explaining15 Sep 2023
- 2023Can we Agree? On the Rash=omon Effect and the Reliability of Post-Hoc Explainable AI14 Aug 2023
- 2023Explainable AI applications in the Medical Domain: a systematic review10 Aug 2023
- 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework24 Jul 2023
- 2023SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction16 Jul 2023
- 2023A Lightweight Generative Model for Interpretable Subject-level Prediction19 Jun 2023
50 newest of 127 All 127 (JSON)
- 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
- 2024Hindsight PRIORs for Reward Learning from Human Preferences12 Apr 2024
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
- 2022Teaching language models to support answers with verified quotescited by 290
- 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
- 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
- 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
- 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
- 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
- 2022Logic-Based Explainability in Machine Learningcited by 47
- 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
- 2022On Tackling Explanation Redundancy in Decision Treescited by 44
- 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
- 2021Tell me why! Explanations support learning relational and causal structurecited by 36
12 most cited of 123 All 123 (JSON)
- 2026Explaining Reinforcement Learning Agents via Inductive Logic Programming15 Jul 2026
- 2026Learning Quantifiable Visual Explanations Without Ground-Truth18 May 2026
- 2026How Can One Choose the Best CAM-Based Explainability Method for a CNN Model?3 May 2026
- 2026Trustworthy Data-driven Chronological Age Estimation from Panoramic Dental Images19 Jan 2026
- 2026Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie18 Jan 2026
- 2025A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts17 Sep 2025
- 2025Neuromorphic Intelligence15 Sep 2025
- 2025DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification6 Aug 2025
- 2025A Survey on Interpretability in Visual Recognition15 Jul 2025
- 2025Gradient Attention Map Based Verification of Deep Convolutional Neural Networks with Application to X-ray Image Datasets29 Apr 2025
- 2025Readable Twins of Unreadable Models17 Apr 2025
- 2025HyConEx: Hypernetwork classifier with counterfactual explanations for tabular data16 Mar 2025
12 newest of 123 All 123 (JSON)