mlsanityData to 7 Oct 2026

Probing Linguistic Systematicity

arXiv 2005.04315 ↗First posted 8 May 2020cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Recently, there has been much interest in the question of whether deep natural language understanding models exhibit systematicity; generalizing such that units like words make consistent contributions to the meaning of the sentences in which they appear. There is accumulating evidence that neural models often generalize non-systematically. We examined the notion of systematicity from a linguistic perspective, defining a set of probes and a set of metrics to measure systematic behaviour. We also identified ways in which network architectures can generalize non-systematically, and discuss why such forms of generalization may be unsatisfying. As a case study, we performed a series of experiments in the setting of natural language inference (NLI), demonstrating that some NLU systems achieve high overall performance despite being non-systematic.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 10; 2023: 6; 2024: 6; 2025: 1; 2026: 2.

37 papers.

Sort:

By area:

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  4. 2021DirectProbe: Studying Representations without Classifierscited by 25
  5. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  6. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  7. 2022Unit Testing for Concepts in Neural Networkscited by 23
  8. 2021Modeling Event Plausibility with Consistent Conceptual Abstractioncited by 20
  9. 2021Structured Reordering for Modeling Latent Alignments in Sequence Transductioncited by 17
  10. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  11. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16
  12. 2021Exploring Transitivity in Neural NLI Models through Veridicalitycited by 15
25 more
  1. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  2. 2023Do PLMs Know and Understand Ontological Knowledge?cited by 13
  3. 2022Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inferencecited by 9
  4. 2022Can Transformers Reason in Fragments of Natural Language?cited by 8
  5. 2021SyGNS: A Systematic Generalization Testbed Based on Natural Language Semanticscited by 7
  6. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blindscited by 7
  7. 2021Supporting Context Monotonicity Abstractions in Neural NLI Modelscited by 5
  8. 2022A Latent-Variable Model for Intrinsic Probingcited by 5
  9. 2022Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspectivecited by 5
  10. 2021Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inferencecited by 4
  11. 2021Decomposing Natural Logic Inferences in Neural NLIcited by 4
  12. 2023Understanding Natural Language Understanding Systems. A Critical Analysiscited by 4
  13. 2023SETI: Systematicity Evaluation of Textual Inferencecited by 4
  14. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  15. 2024Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammarscited by 3
  16. 2021Situation and Behavior Understanding by Trope Detection on Filmscited by 2
  17. 2022Do Transformers know symbolic rules, and would we know if they did?cited by 1
  18. 2023Dynamic MOdularized Reasoning for Compositional Structured Explanation Generationcited by 1
  19. 2024DeepCodeProbe: Towards Understanding What Models Trained on Code Learncited by 1
  20. 2023Systematic word meta-sense extensioncited by 0
  21. 2024EntailE: Introducing Textual Entailment in Commonsense Knowledge Graph Completioncited by 0
  22. 2024Empirical Study of Symmetrical Reasoning in Conversational Chatbotscited by 0
  23. 2025Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Modelscited by 0
  24. 2026Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Modelscited by 0
  25. 2026Revisiting the Systematicity in Negation in the Era of In-Context Learningcited by 0