# From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP

arXiv: https://arxiv.org/abs/2406.12618 · HTML: https://mlsanity.com/paper/2406.12618/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 68 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://mlsanity.com/paper/1904.01608/) (2019)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Examining Citations of Natural Language Processing Literature](https://mlsanity.com/paper/2005.00912/) (2020)
- [We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields](https://mlsanity.com/paper/2310.14870/) (2023)
- [A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?](https://mlsanity.com/paper/2305.12920/) (2023)
- [Forgotten Knowledge: Examining the Citational Amnesia in NLP](https://mlsanity.com/paper/2305.18554/) (2023)
- [Measuring academic influence: Not all citations are equal](https://mlsanity.com/paper/1501.06587/) (2015)

## Built on by (most cited first, 20 of 20)

- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark](https://mlsanity.com/paper/2601.08331/) (2026)
- [What Do Speech Foundation Models Not Learn About Speech?](https://mlsanity.com/paper/2410.12948/) (2024)
- [Interpretability Can Be Actionable](https://mlsanity.com/paper/2605.11161/) (2026)
- [Aligned Probing: Relating Toxic Behavior and Model Internals](https://mlsanity.com/paper/2503.13390/) (2025)
- [Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement](https://mlsanity.com/paper/2505.23183/) (2025)
- [Challenges in Understanding Modality Conflict in Vision-Language Models](https://mlsanity.com/paper/2509.02805/) (2025)
- [Tracr-Injection: Distilling Algorithms into Pre-trained Language Models](https://mlsanity.com/paper/2505.10719/) (2025)
- [ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.24388/) (2025)
- [Risks of AI-driven product development and strategies for their mitigation](https://mlsanity.com/paper/2506.00047/) (2025)
- [Good Intentions Beyond ACL: Who Does NLP for Social Good, and Where?](https://mlsanity.com/paper/2510.04434/) (2025)
- [Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models](https://mlsanity.com/paper/2510.13580/) (2025)
- [From Explainability to Action: A Generative Operational Framework for Integrating XAI in Clinical Mental Health Screening](https://mlsanity.com/paper/2510.13828/) (2025)
- [Diffusion-CAM: Faithful Visual Explanations for dMLLMs](https://mlsanity.com/paper/2604.11005/) (2026)
- [Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors](https://mlsanity.com/paper/2502.19669/) (2025)
- [Can Gradient Descent Simulate Prompting?](https://mlsanity.com/paper/2506.20989/) (2025)
- [NLP Meets the World: Toward Improving Conversations With the Public About Natural Language Processing Research](https://mlsanity.com/paper/2507.10559/) (2025)
- [Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms](https://mlsanity.com/paper/2604.15842/) (2026)
- [The Geometry of Low-Resource Language Representations](https://mlsanity.com/paper/2608.23358/) (2026)

## Abstract (arXiv)

Interpretability and analysis (IA) research is a growing subfield within NLP with the goal of developing a deeper understanding of the behavior or inner workings of NLP systems and methods. Despite growing interest in the subfield, a criticism of this work is that it lacks actionable insights and therefore has little impact on NLP. In this paper, we seek to quantify the impact of IA research on the broader field of NLP. We approach this with a mixed-methods analysis of: (1) a citation graph of 185K+ papers built from all papers published at ACL and EMNLP conferences from 2018 to 2023, and their references and citations, and (2) a survey of 138 members of the NLP community. Our quantitative results show that IA work is well-cited outside of IA, and central in the NLP citation graph. Through qualitative analysis of survey responses and manual annotation of 556 papers, we find that NLP researchers build on findings from IA work and perceive it as important for progress in NLP, multiple subfields, and rely on its findings and terminology for their own work. Many novel methods are proposed based on IA findings and highly influenced by them, but highly influential non-IA work cites IA findings without being driven by them. We end by summarizing what is missing in IA work today and provide a call to action, to pave the way for a more impactful future of IA research.