# An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models

arXiv: https://arxiv.org/abs/2004.02451 · HTML: https://mlsanity.com/paper/2004.02451/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 7 of 7)

- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)
- [Learning with Instance Bundles for Reading Comprehension](https://mlsanity.com/paper/2104.08735/) (2021)
- [A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers](https://mlsanity.com/paper/2507.22337/) (2025)
- [Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion](https://mlsanity.com/paper/2204.08644/) (2022)
- [Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency](https://mlsanity.com/paper/2404.09830/) (2024)
- [Structural Automata for Neural Semantic Parsing](https://mlsanity.com/paper/2610.13896/) (2026)

## Abstract (arXiv)

We explore the utilities of explicit negative examples in training neural language models. Negative examples here are incorrect words in a sentence, such as "barks" in "*The dogs barks". Neural language models are commonly trained only on positive examples, a set of sentences in the training data, but recent studies suggest that the models trained in this way are not capable of robustly handling complex syntactic constructions, such as long-distance agreement. In this paper, using English data, we first demonstrate that appropriately using negative examples about particular constructions (e.g., subject-verb agreement) will boost the model's robustness on them, with a negligible loss of perplexity. The key to our success is an additional margin loss between the log-likelihoods of a correct word and an incorrect word. We then provide a detailed analysis of the trained models. One of our findings is the difficulty of object-relative clauses for RNNs. We find that even with our direct learning signals the models still suffer from resolving agreement across an object-relative clause. Augmentation of training sentences involving the constructions somewhat helps, but the accuracy still does not reach the level of subject-relative clauses. Although not directly cognitively appealing, our method can be a tool to analyze the true architectural limitation of neural models on challenging linguistic constructions.