# Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?

arXiv: https://arxiv.org/abs/2004.14839 · HTML: https://mlsanity.com/paper/2004.14839/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 29 of 29)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Figurative Language in Recognizing Textual Entailment](https://mlsanity.com/paper/2106.01195/) (2021)
- [Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability](https://mlsanity.com/paper/2112.09054/) (2021)
- [RECKONING: Reasoning through Dynamic Knowledge Encoding](https://mlsanity.com/paper/2305.06349/) (2023)
- [Exploring Transitivity in Neural NLI Models through Veridicality](https://mlsanity.com/paper/2101.10713/) (2021)
- [Scope Ambiguities in Large Language Models](https://mlsanity.com/paper/2404.04332/) (2024)
- [Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference](https://mlsanity.com/paper/2203.04857/) (2022)
- [Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks](https://mlsanity.com/paper/2204.10615/) (2022)
- [Can Transformers Reason in Fragments of Natural Language?](https://mlsanity.com/paper/2211.05417/) (2022)
- [Representation Of Lexical Stylistic Features In Language Models' Embedding Space](https://mlsanity.com/paper/2305.18657/) (2023)
- [SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics](https://mlsanity.com/paper/2106.01077/) (2021)
- [Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds](https://mlsanity.com/paper/2305.14785/) (2023)
- [Do Grammatical Error Correction Models Realize Grammatical Generalization?](https://mlsanity.com/paper/2106.03031/) (2021)
- [Supporting Context Monotonicity Abstractions in Neural NLI Models](https://mlsanity.com/paper/2105.08008/) (2021)
- [SETI: Systematicity Evaluation of Textual Inference](https://mlsanity.com/paper/2305.15045/) (2023)
- [SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space](https://mlsanity.com/paper/2307.02269/) (2023)
- [Exploring Continual Learning of Compositional Generalization in NLI](https://mlsanity.com/paper/2403.04400/) (2024)
- [NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic](https://mlsanity.com/paper/2307.02849/) (2023)
- [Collaborative Development of NLP models](https://mlsanity.com/paper/2305.12219/) (2023)
- [The Case for Scalable, Data-Driven Theory: A Paradigm for Scientific Progress in NLP](https://mlsanity.com/paper/2312.00349/) (2023)
- [Reinforcement Learning for Compositional Generalization with Outcome-Level Optimization](https://mlsanity.com/paper/2605.04920/) (2026)
- [Dynamic MOdularized Reasoning for Compositional Structured Explanation Generation](https://mlsanity.com/paper/2309.07624/) (2023)
- [Formal Proofs as Structured Explanations: Proposing Several Tasks on Explainable Natural Language Inference](https://mlsanity.com/paper/2311.08637/) (2023)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Systematic word meta-sense extension](https://mlsanity.com/paper/2311.13029/) (2023)
- [Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models](https://mlsanity.com/paper/2508.17153/) (2025)
- [Revisiting the Systematicity in Negation in the Era of In-Context Learning](https://mlsanity.com/paper/2606.16867/) (2026)

## Abstract (arXiv)

Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. In this paper, we introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language, namely, the regularity for performing arbitrary inferences with generalization on composition. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits. A series of experiments show that three neural models systematically draw inferences on unseen combinations of lexical and logical phenomena when the syntactic structures of the sentences are similar between the training and test sets. However, the performance of the models significantly decreases when the structures are slightly changed in the test set while retaining all vocabularies and constituents already appearing in the training set. This indicates that the generalization ability of neural models is limited to cases where the syntactic structures are nearly the same as those in the training set.