mlsanityData to 7 Oct 2026

Understanding the Role of Individual Units in a Deep Neural Network

arXiv 2009.05041 ↗First posted 10 Sep 2020cs.CV cs.LG cs.NEcited by 254

Not on the map. Why?

Abstract (arXiv)

Deep neural networks excel at finding hierarchical representations that solve complex tasks over large data sets. How can we humans understand these learned representations? In this work, we present network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks. First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts. We find evidence that the network has learned many object classes that play crucial roles in classifying scene classes. Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes. By analyzing changes made when small sets of units are activated or deactivated, we find that objects can be added and removed from the output scenes while adapting to the context. Finally, we apply our analytic framework to understanding adversarial attacks and to semantic image editing.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 44; 2023: 54; 2024: 50; 2025: 47; 2026: 20.

254 papers.

Sort:

By area:

  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  2. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  3. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  4. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  5. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  6. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  7. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 144
  8. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  9. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  10. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  11. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  12. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
38 more
  1. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  2. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  3. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  4. 2024Universal Neurons in GPT2 Language Modelscited by 100
  5. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  6. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  7. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  8. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 84
  9. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  10. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  11. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  12. 2021Editing a classifier by rewriting its prediction rulescited by 64
  13. 2024A Multimodal Automated Interpretability Agentcited by 62
  14. 2023Explaining black box text modules in natural language with language modelscited by 61
  15. 2022Third Time's the Charm? Image and Video Editing with StyleGAN3cited by 54
  16. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  17. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  18. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  19. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  20. 2024Do language models plan ahead for future tokens?cited by 47
  21. 2023Adaptive Frequency Filters As Efficient Global Token Mixerscited by 45
  22. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  23. 2023StyleGAN knows Normal, Depth, Albedo, and Morecited by 44
  24. 2022Rethinking the Reverse-engineering of Trojan Triggerscited by 37
  25. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
  26. 2024Interpreting the Second-Order Effects of Neurons in CLIPcited by 36
  27. 2024Explorations of Self-Repair in Language Modelscited by 35
  28. 2023LINe: Out-of-Distribution Detection by Leveraging Important Neuronscited by 34
  29. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  30. 2022DL4SciVis: A State-of-the-Art Survey on Deep Learning for Scientific Visualizationcited by 32
  31. 2023Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitmentcited by 32
  32. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
  33. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  34. 2024Language Models Represent Beliefs of Self and Otherscited by 29
  35. 2023Disentangling Neuron Representations with Concept Vectorscited by 28
  36. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  37. 2025Training Language Models to Explain Their Own Computationscited by 27
  38. 2022Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Valuescited by 25

50 most cited of 254 All 254 (JSON)