# Discovery of Natural Language Concepts in Individual Units of CNNs

arXiv: https://arxiv.org/abs/1902.07249 · HTML: https://mlsanity.com/paper/1902.07249/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2019.

## Built on by (most cited first, 21 of 21)

- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [Configurable Foundation Models: Building LLMs from a Modular Perspective](https://mlsanity.com/paper/2409.02877/) (2024)
- [NeuroStrike: Neuron-Level Attacks on Aligned LLMs](https://mlsanity.com/paper/2509.11864/) (2025)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [Towards a fuller understanding of neurons with Clustered Compositional Explanations](https://mlsanity.com/paper/2310.18443/) (2023)
- [From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks](https://mlsanity.com/paper/2310.11884/) (2023)
- [Evaluating Readability and Faithfulness of Concept-based Explanations](https://mlsanity.com/paper/2404.18533/) (2024)
- [Evaluating Neuron Interpretation Methods of NLP Models](https://mlsanity.com/paper/2301.12608/) (2023)
- [On the special role of class-selective neurons in early training](https://mlsanity.com/paper/2305.17409/) (2023)
- [Fine-grained Interpretation and Causation Analysis in Deep NLP Models](https://mlsanity.com/paper/2105.08039/) (2021)
- [Investigating the Encoding of Words in BERT's Neurons using Feature Textualization](https://mlsanity.com/paper/2311.08240/) (2023)
- [Manipulating Feature Visualizations with Gradient Slingshots](https://mlsanity.com/paper/2401.06122/) (2024)
- [NeST: Neuron Selective Tuning for LLM Safety](https://mlsanity.com/paper/2602.16835/) (2026)
- [NxPlain: Web-based Tool for Discovery of Latent Concepts](https://mlsanity.com/paper/2303.03019/) (2023)
- [Towards Generating Informative Textual Description for Neurons in Language Models](https://mlsanity.com/paper/2401.16731/) (2024)
- [Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents](https://mlsanity.com/paper/2409.16934/) (2024)

## Abstract (arXiv)

Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret. Especially, little is known about how they represent language in their intermediate layers. In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to specific morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns. In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text. We conduct analyses with different architectures on multiple datasets for classification and translation tasks and provide new insights into how deep models understand natural language.