# Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda

arXiv: https://arxiv.org/abs/2404.09554 · HTML: https://mlsanity.com/paper/2404.09554/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 136 of its 223 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 30 of 136)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)

## Built on by (most cited first, 22 of 22)

- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [Generative Models for Crystalline Materials](https://mlsanity.com/paper/2511.22652/) (2025)
- [LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation](https://mlsanity.com/paper/2506.11476/) (2025)
- [AgentFoX: LLM Agent-Guided Fusion with eXplainability for AI-Generated Image Detection](https://mlsanity.com/paper/2603.23115/) (2026)
- [Topic Modeling with Fine-tuning LLMs and Bag of Sentences](https://mlsanity.com/paper/2408.03099/) (2024)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors](https://mlsanity.com/paper/2603.02792/) (2026)
- [Improving Next Tokens via Second-to-Last Predictions with Generate and Refine](https://mlsanity.com/paper/2411.15661/) (2024)
- [What's Wrong with Your Synthetic Tabular Data? Using Explainable AI to Evaluate Generative Models](https://mlsanity.com/paper/2504.20687/) (2025)
- [Generative AI and the future of scientometrics: current topics and future questions](https://mlsanity.com/paper/2507.00783/) (2025)
- [AttentionBender: Manipulating Cross-Attention in Video Diffusion Transformers as a Creative Probe](https://mlsanity.com/paper/2604.20936/) (2026)
- [Using Phonemes in cascaded S2S translation pipeline](https://mlsanity.com/paper/2504.16234/) (2025)
- [Enhancing ML Model Interpretability: Leveraging Fine-Tuned Large Language Models for Better Understanding of AI](https://mlsanity.com/paper/2505.02859/) (2025)
- [Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only data](https://mlsanity.com/paper/2506.23182/) (2025)
- [Interpretability Framework for LLMs in Undergraduate Calculus](https://mlsanity.com/paper/2510.17910/) (2025)
- [Large Language Models for Explainable Threat Intelligence](https://mlsanity.com/paper/2511.05406/) (2025)
- [Emergent, not Immanent: A Baradian Reading of Explainable AI](https://mlsanity.com/paper/2601.15029/) (2026)
- [Stochastic Neural Networks for Quantum Devices](https://mlsanity.com/paper/2602.22241/) (2026)
- [Addressing the Selection Problem in Explainable AI](https://mlsanity.com/paper/2608.22356/) (2026)
- [A Comprehensive Review of Generative Physical Artificial Intelligence](https://mlsanity.com/paper/2609.18111/) (2026)

## Abstract (arXiv)

Generative AI (GenAI) marked a shift from AI being able to recognize to AI being able to generate solutions for a wide variety of tasks. As the generated solutions and applications become increasingly more complex and multi-faceted, novel needs, objectives, and possibilities have emerged for explainability (XAI). In this work, we elaborate on why XAI has gained importance with the rise of GenAI and its challenges for explainability research. We also unveil novel and emerging desiderata that explanations should fulfill, covering aspects such as verifiability, interactivity, security, and cost. To this end, we focus on surveying existing works. Furthermore, we provide a taxonomy of relevant dimensions that allows us to better characterize existing XAI mechanisms and methods for GenAI. We discuss different avenues to ensure XAI, from training data to prompting. Our paper offers a short but concise technical background of GenAI for non-technical readers, focusing on text and images to better understand novel or adapted XAI techniques for GenAI. However, due to the vast array of works on GenAI, we decided to forego detailed aspects of XAI related to evaluation and usage of explanations. As such, the manuscript interests both technically oriented people and other disciplines, such as social scientists and information systems researchers. Our research roadmap provides more than ten directions for future investigation.