mlsanityData to 7 Oct 2026

Investigating Explainability of Generative AI for Code through Scenario-based Design

arXiv 2202.04903 ↗First posted 10 Feb 2022cs.HC cs.AI cs.SEcited by 47

Not on the map. Why?

Abstract (arXiv)

What does it mean for a generative AI model to be explainable? The emergent discipline of explainable AI (XAI) has made great strides in helping people understand discriminative models. Less attention has been paid to generative models that produce artifacts, rather than decisions, as output. Meanwhile, generative AI (GenAI) technologies are maturing and being applied to application domains such as software engineering. Using scenario-based design and question-driven XAI design approaches, we explore users' explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion. We conducted 9 workshops with 43 software engineers in which real examples from state-of-the-art generative AI models were used to elicit users' explainability needs. Drawing from prior work, we also propose 4 types of XAI features for GenAI for code and gathered additional design ideas from participants. Our work explores explainability needs for GenAI for code and demonstrates how human-centered approaches can drive the technical development of XAI in novel domains.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 50 of its 108 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

50 of 108 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2018Model Cards for Model Reportingcited by 1,068
  11. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
38 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2016The Mythos of Model Interpretabilitycited by 716
  4. 2019Attention is not not Explanationcited by 629
  5. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  6. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  7. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  8. 2016Rationalizing Neural Predictionscited by 281
  9. 2020Unsupervised Translation of Programming Languagescited by 262
  10. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  11. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  12. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  13. 2019Explainable Machine Learning in Deploymentcited by 210
  14. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 128
  15. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  16. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  17. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  18. 2018Learning Deep Disentangled Embeddings with the F-Statistic Losscited by 101
  19. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  20. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  21. 2020The Pragmatic Turn in Explainable Artificial Intelligence (XAI)cited by 53
  22. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42
  23. 2019Human-Centered Artificial Intelligence and Machine Learningcited by 37
  24. 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
  25. 2018FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformitycited by 31
  26. 2021Question-Driven Design Process for Explainable AI User Experiencescited by 31
  27. 2021How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Studycited by 29
  28. 2016A Survey of Inductive Biases for Factorial Representation-Learningcited by 23
  29. 2021Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AIcited by 22
  30. 2021The Sanction of Authority: Promoting Public Trust in AIcited by 19
  31. 2021Towards A Process Model for Co-Creating AI Experiencescited by 18
  32. 2021Evaluating the Interpretability of Generative Models by Interactive Reconstructioncited by 16
  33. 2020A Methodology for Creating AI FactSheetscited by 15
  34. 2019Experiences with Improving the Transparency of AI Models and Servicescited by 14
  35. 2019ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecyclescited by 13
  36. 2019Enabling Value Sensitive AI Systems through Participatory Design Fictionscited by 11
  37. 2020Towards evaluating and eliciting high-quality documentation for intelligent systemscited by 7
  38. 2020Quality Estimation & Interpretability for Code Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 14; 2024: 15; 2025: 12; 2026: 4.

47 papers.

Sort:

By area:

  1. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  2. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  3. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  4. 2024Design Principles for Generative AI Applicationscited by 57
  5. 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
  6. 2023Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAIcited by 33
  7. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  8. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  9. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  10. 2024Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seekingcited by 24
  11. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  12. 2023Toward General Design Principles for Generative AI Applicationscited by 20
35 more
  1. 2023Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completionscited by 20
  2. 2024Using LLMs in Software Requirements Specifications: An Empirical Evaluationcited by 20
  3. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  4. 2023Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Toolscited by 16
  5. 2023"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Modelscited by 15
  6. 2023Next Steps for Human-Centered Generative AI: A Technical Perspectivecited by 13
  7. 2024Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seekingcited by 13
  8. 2025Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creationcited by 13
  9. 2024Exploring Prompt Engineering Practices in the Enterprisecited by 11
  10. 2023R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intentscited by 10
  11. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  12. 2022Requirements Engineering for Machine Learning: A Review and Reflectioncited by 7
  13. 2023Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytellingcited by 7
  14. 2024Large Language Models for the Automated Analysis of Optimization Algorithmscited by 5
  15. 2024How Human-Centered Explainable AI Interface Are Designed and Evaluated: A Systematic Surveycited by 5
  16. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  17. 2025Statistical Methods in Generative AIcited by 4
  18. 2025Less or More: Towards Glanceable Explanations for LLM Recommendations Using Ultra-Small Devicescited by 3
  19. 2026Humans are Missing from AI Coding Agent Researchcited by 3
  20. 2023Explaining CLIP through Co-Creative Drawings and Interactioncited by 2
  21. 2024The European Commitment to Human-Centered Technology: The Integral Role of HCI in the EU AI Act's Successcited by 2
  22. 2024Governance of Generative Artificial Intelligence for Companiescited by 2
  23. 2024The AI-DEC: A Card-based Design Method for User-centered AI Explanationscited by 2
  24. 2025Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Reviewcited by 2
  25. 2025Scenarios in Computing Research: A Systematic Review of the Use of Scenario Methods for Exploring the Future of Computing Technologies in Societycited by 2
  26. 2024Generative AI in Medicinecited by 1
  27. 2025HiLDe: Intentional Code Generation via Human-in-the-Loop Decodingcited by 1
  28. 2025Hide or Highlight: Understanding the Impact of Factuality Expression on User Trustcited by 1
  29. 2023Human-AI Co-Creation Approach to Find Forever Chemicals Replacementscited by 0
  30. 2024Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Modelscited by 0
  31. 2025Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agentscited by 0
  32. 2025MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systemscited by 0
  33. 2026LLM Nepotism in Organizational Governancecited by 0
  34. 2026Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraphcited by 0
  35. 2026Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Reviewcited by 0