mlsanityData to 7 Oct 2026

A Systematic Literature Review on Explainability for Machine/Deep Learning-based Software Engineering Research

arXiv 2401.14617 ↗First posted 26 Jan 2024cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

The remarkable achievements of Artificial Intelligence (AI) algorithms, particularly in Machine Learning (ML) and Deep Learning (DL), have fueled their extensive deployment across multiple sectors, including Software Engineering (SE). However, due to their black-box nature, these promising AI-driven SE models are still far from being deployed in practice. This lack of explainability poses unwanted risks for their applications in critical tasks, such as vulnerability detection, where decision-making transparency is of paramount importance. This paper endeavors to elucidate this interdisciplinary domain by presenting a systematic literature review of approaches that aim to improve the explainability of AI models within the context of SE. The review canvasses work appearing in the most prominent SE & AI conferences and journals, and spans 108 papers across 23 unique SE tasks. Based on three key Research Questions (RQs), we aim to (1) summarize the SE tasks where XAI techniques have shown success to date; (2) classify and analyze different XAI techniques; and (3) investigate existing evaluation approaches. Based on our findings, we identified a set of challenges remaining to be addressed in existing studies, together with a set of guidelines highlighting potential opportunities we deemed appropriate and important for future work.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 53 of its 129 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

53 of 129 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Graph Attention Networkscited by 8,854
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  9. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  10. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  11. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  12. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
38 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  3. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  5. 2019GNNExplainer: Generating Explanations for Graph Neural Networkscited by 735
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  8. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  9. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  10. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248
  11. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  12. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  13. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  14. 2019explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learningcited by 69
  15. 2021Vulnerability Detection with Fine-grained Interpretationscited by 56
  16. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  17. 2020A Survey on Deep Learning for Software Engineeringcited by 37
  18. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  19. 2020A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Imagescited by 35
  20. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 31
  21. 2021Smart Contract Vulnerability Detection: From Pure Neural Network to Interpretable Graph Feature and Expert Pattern Fusioncited by 24
  22. 2021Counterfactual Explanations for Models of Codecited by 24
  23. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  24. 2021Understanding Neural Code Intelligence Through Program Simplificationcited by 20
  25. 2022SoK: Explainable Machine Learning for Computer Security Applicationscited by 20
  26. 2022Actionable and Interpretable Fault Localization for Recurring Failures in Online Service Systemscited by 18
  27. 2018Explanations of model predictions with live and breakDown packagescited by 17
  28. 2020Why an Android App is Classified as Malware? Towards Malware Classification Interpretationcited by 16
  29. 2018Explainable Software Analyticscited by 15
  30. 2020Predicting Defective Lines Using a Model-Agnostic Techniquecited by 14
  31. 2023Benchmarking Causal Study to Interpret Large Language Models for Source Codecited by 14
  32. 2023Distinguishing Look-Alike Innocent and Vulnerable Code by Subtle Semantic Representation Learning and Explanationcited by 11
  33. 2022Explainable AI for Android Malware Detection: Towards Understanding Why the Models Perform So Well?cited by 10
  34. 2023A Comprehensive Review on Financial Explainable AIcited by 10
  35. 2021Mate! Are You Really Aware? An Explainability-Guided Testing Framework for Robustness of Malware Detectorscited by 7
  36. 2023TraceDiag: Adaptive, Interpretable, and Efficient Root Cause Analysis on Large-Scale Microservice Systemscited by 7
  37. 2019STYLE-ANALYZER: fixing code style inconsistencies with interpretable unsupervised algorithmscited by 6
  38. 2023Silent Vulnerable Dependency Alert Prediction with Vulnerability Key Aspect Explanationcited by 5

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Interacting with AI Reasoning Models: Harnessing "Thoughts" for AI-Driven Software Engineeringcited by 2
  2. 2024A Systematic Literature Review on the Use of Machine Learning in Software Engineeringcited by 0