# Negation, Coordination, and Quantifiers in Contextualized Language Models

arXiv: https://arxiv.org/abs/2209.07836 · HTML: https://mlsanity.com/paper/2209.07836/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Probing Natural Language Inference Models through Semantic Fragments](https://mlsanity.com/paper/1909.07521/) (2019)
- [Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2201.12438/) (2022)
- [Stopwords in Technical Language Processing](https://mlsanity.com/paper/2006.02633/) (2020)
- ["Not not bad" is not "bad": A distributional account of negation](https://mlsanity.com/paper/1306.2158/) (2013)
- [Prior matters: simple and general methods for evaluating and improving topic quality in topic modeling](https://mlsanity.com/paper/1701.03227/) (2017)

## Built on by (most cited first, 9 of 9)

- [Limits for Learning with Language Models](https://mlsanity.com/paper/2306.12213/) (2023)
- [SyntaxShap: Syntax-aware Explainability Method for Text Generation](https://mlsanity.com/paper/2402.09259/) (2024)
- [Negation: A Pink Elephant in the Large Language Models' Room?](https://mlsanity.com/paper/2503.22395/) (2025)
- [Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges](https://mlsanity.com/paper/2310.11252/) (2023)
- [Probing Quantifier Comprehension in Large Language Models: Another Example of Inverse Scaling](https://mlsanity.com/paper/2306.07384/) (2023)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers](https://mlsanity.com/paper/2212.08700/) (2022)
- [Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models](https://mlsanity.com/paper/2312.11720/) (2023)
- [Revisiting the Systematicity in Negation in the Era of In-Context Learning](https://mlsanity.com/paper/2606.16867/) (2026)

## Abstract (arXiv)

With the success of contextualized language models, much research explores what these models really learn and in which cases they still fail. Most of this work focuses on specific NLP tasks and on the learning outcome. Little research has attempted to decouple the models' weaknesses from specific tasks and focus on the embeddings per se and their mode of learning. In this paper, we take up this research opportunity: based on theoretical linguistic insights, we explore whether the semantic constraints of function words are learned and how the surrounding context impacts their embeddings. We create suitable datasets, provide new insights into the inner workings of LMs vis-a-vis function words and implement an assisting visual web interface for qualitative analysis.