# Interpretable Textual Neuron Representations for NLP

arXiv: https://arxiv.org/abs/1809.07291 · HTML: https://mlsanity.com/paper/1809.07291/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 16 of 16)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Neuron to Graph: Interpreting Language Model Neurons at Scale](https://mlsanity.com/paper/2305.19911/) (2023)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [Evaluating Neuron Interpretation Methods of NLP Models](https://mlsanity.com/paper/2301.12608/) (2023)
- [Examining the Causal Effect of First Names on Language Models: The Case of Social Commonsense Reasoning](https://mlsanity.com/paper/2306.01117/) (2023)
- [Fluent dreaming for language models](https://mlsanity.com/paper/2402.01702/) (2024)
- [Fine-grained Interpretation and Causation Analysis in Deep NLP Models](https://mlsanity.com/paper/2105.08039/) (2021)
- [N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models](https://mlsanity.com/paper/2304.12918/) (2023)
- [Investigating the Encoding of Words in BERT's Neurons using Feature Textualization](https://mlsanity.com/paper/2311.08240/) (2023)
- [Explaining Any ML Model? -- On Goals and Capabilities of XAI](https://mlsanity.com/paper/2206.13888/) (2022)
- [NxPlain: Web-based Tool for Discovery of Latent Concepts](https://mlsanity.com/paper/2303.03019/) (2023)
- [ContextBench: Modifying Contexts for Targeted Latent Activation](https://mlsanity.com/paper/2506.15735/) (2025)

## Abstract (arXiv)

Input optimization methods, such as Google Deep Dream, create interpretable representations of neurons for computer vision DNNs. We propose and evaluate ways of transferring this technology to NLP. Our results suggest that gradient ascent with a gumbel softmax layer produces n-gram representations that outperform naive corpus search in terms of target neuron activation. The representations highlight differences in syntax awareness between the language and visual models of the Imaginet architecture.