mlsanityData to 7 Oct 2026

Unit Testing for Concepts in Neural Networks

arXiv 2208.10244 ↗First posted 28 Jul 2022cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of "cat" is related to our concepts of "ears" and "whiskers" in a non-arbitrary way. Fodor (1998) proposes one theory of concepts, which emphasizes symbolic representations related via constituency structures. Whether neural networks are consistent with such a theory is open for debate. We propose unit tests for evaluating whether a system's behavior is consistent with several key aspects of Fodor's criteria. Using a simple visual concept learning task, we evaluate several modern neural architectures against this specification. We find that models succeed on tests of groundedness, modularlity, and reusability of concepts, but that important questions about causality remain open. Resolving these will require new methods for analyzing models' internal states.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 28 Jul 2022.

Builds on

36 of 63 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2019Attention is not not Explanationcited by 629
  12. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
24 more
  1. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  2. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  3. 2015Neural Module Networkscited by 375
  4. 2019Counterfactual Visual Explanationscited by 312
  5. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  6. 2013OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networkscited by 224
  7. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  8. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  9. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  10. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  11. 2020Debugging Tests for Model Explanationscited by 146
  12. 2018Explaining Image Classifiers by Counterfactual Generationcited by 144
  13. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  14. 2020Syntactic Structure from Deep Learningcited by 99
  15. 2019Measuring Compositionality in Representation Learningcited by 84
  16. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  17. 2019Distributional Semantics and Linguistic Theorycited by 53
  18. 2019Can neural networks understand monotonicity reasoning?cited by 45
  19. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  20. 2020Probing Linguistic Systematicitycited by 37
  21. 2019Correlating neural and symbolic representations of languagecited by 32
  22. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  23. 2020Amnesic Probing: Behavioral Explanation with Amnesic Counterfactualscited by 12
  24. 2020Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)cited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 9; 2024: 6; 2025: 3; 2026: 1.

23 papers.

Sort:

By area:

  1. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  2. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  3. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  4. 2023Eight Things to Know about Large Language Modelscited by 73
  5. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  6. 2023The Hidden Language of Diffusion Modelscited by 34
  7. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  8. 2023Dissociating language and thought in large language modelscited by 27
  9. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  10. 2022Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projectioncited by 17
  11. 2024Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?cited by 17
  12. 2023Operationalising Representation in Natural Language Processingcited by 14
11 more
  1. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  2. 2023Angler: Helping Machine Translation Practitioners Prioritize Model Improvementscited by 11
  3. 2024Towards Compositionality in Concept Learningcited by 7
  4. 2024On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probecited by 6
  5. 2025Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventionscited by 6
  6. 2022Language Models Understand Us, Poorlycited by 3
  7. 2023Deep Natural Language Feature Learning for Interpretable Predictioncited by 1
  8. 2024Interpreting the structure of multi-object representations in vision encoderscited by 1
  9. 2025On the Performance of Concept Probing: The Influence of the Data (Extended Version)cited by 0
  10. 2025Concept Probing: Where to Find Human-Defined Concepts (Extended Version)cited by 0
  11. 2026Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Groundingcited by 0