# Using Captum to Explain Generative Language Models

arXiv: https://arxiv.org/abs/2312.05491 · HTML: https://mlsanity.com/paper/2312.05491/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [OpenAttack: An Open-source Textual Adversarial Attack Toolkit](https://mlsanity.com/paper/2009.09191/) (2020)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Data Augmentation for Neural NLP](https://mlsanity.com/paper/2302.11412/) (2023)

## Built on by (most cited first, 30 of 33)

- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [Understanding the Dark Side of LLMs' Intrinsic Self-Correction](https://mlsanity.com/paper/2412.14959/) (2024)
- [Backward Lens: Projecting Language Model Gradients into the Vocabulary Space](https://mlsanity.com/paper/2402.12865/) (2024)
- [Navigating LLM Ethics: Advancements, Challenges, and Future Directions](https://mlsanity.com/paper/2406.18841/) (2024)
- [LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models](https://mlsanity.com/paper/2404.07004/) (2024)
- [CLEAR: Can Language Models Really Understand Causal Graphs?](https://mlsanity.com/paper/2406.16605/) (2024)
- [Learning to Understand: Identifying Interactions via the M"obius Transform](https://mlsanity.com/paper/2402.02631/) (2024)
- [Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns](https://mlsanity.com/paper/2404.14812/) (2024)
- [TracLLM: A Generic Framework for Attributing Long Context LLMs](https://mlsanity.com/paper/2506.04202/) (2025)
- [SyntaxShap: Syntax-aware Explainability Method for Text Generation](https://mlsanity.com/paper/2402.09259/) (2024)
- [Multi-Level Explanations for Generative Language Models](https://mlsanity.com/paper/2403.14459/) (2024)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations](https://mlsanity.com/paper/2402.12038/) (2024)
- [Towards Achieving Concept Completeness for Textual Concept Bottleneck Models](https://mlsanity.com/paper/2502.11100/) (2025)
- [AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruption](https://mlsanity.com/paper/2508.03793/) (2025)
- [Interactive Prompt Debugging with Sequence Salience](https://mlsanity.com/paper/2404.07498/) (2024)
- [JoPA:Explaining Large Language Model's Generation via Joint Prompt Attribution](https://mlsanity.com/paper/2405.20404/) (2024)
- [ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMs](https://mlsanity.com/paper/2410.17406/) (2024)
- [SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation](https://mlsanity.com/paper/2411.01710/) (2024)
- [DUE: Dynamic Uncertainty-Aware Explanation Supervision via 3D Imputation](https://mlsanity.com/paper/2403.10831/) (2024)
- [Understanding the Dilemma of Unlearning for Large Language Models](https://mlsanity.com/paper/2509.24675/) (2025)
- [Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Models](https://mlsanity.com/paper/2408.11252/) (2024)
- [SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling](https://mlsanity.com/paper/2508.09105/) (2025)
- [Selective Explanations](https://mlsanity.com/paper/2405.19562/) (2024)
- [Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMs](https://mlsanity.com/paper/2409.13764/) (2024)
- [EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models](https://mlsanity.com/paper/2505.01238/) (2025)
- [Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits](https://mlsanity.com/paper/2602.00092/) (2026)
- [Effective Large Language Model Debugging with Best-first Tree Search](https://mlsanity.com/paper/2407.19055/) (2024)
- [Document Attribution: Examining Citation Relationships using Large Language Models](https://mlsanity.com/paper/2505.06324/) (2025)
- [Identifying Good and Bad Neurons for Task-Level Controllable LLMs](https://mlsanity.com/paper/2601.04548/) (2026)

## Abstract (arXiv)

Captum is a comprehensive library for model explainability in PyTorch, offering a range of methods from the interpretability literature to enhance users' understanding of PyTorch models. In this paper, we introduce new features in Captum that are specifically designed to analyze the behavior of generative language models. We provide an overview of the available functionalities and example applications of their potential for understanding learned associations within generative language models.