# Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA

arXiv: https://arxiv.org/abs/2009.08257 · HTML: https://mlsanity.com/paper/2009.08257/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 12 of 12)

- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task](https://mlsanity.com/paper/2306.15298/) (2023)
- [XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge](https://mlsanity.com/paper/2109.12573/) (2021)
- [GCDT: A Chinese RST Treebank for Multigenre and Multilingual Discourse Parsing](https://mlsanity.com/paper/2210.10449/) (2022)
- [John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs](https://mlsanity.com/paper/2106.01060/) (2021)
- [A Community-Centric Perspective for Characterizing and Detecting Anti-Asian Violence-Provoking Speech](https://mlsanity.com/paper/2407.15227/) (2024)
- [A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers](https://mlsanity.com/paper/2507.22337/) (2025)
- [Quantifying the Task-Specific Information in Text-Based Classifications](https://mlsanity.com/paper/2110.08931/) (2021)
- [Explaining Search Result Stances to Opinionated People](https://mlsanity.com/paper/2309.08460/) (2023)
- [αVIL: Learning to Leverage Auxiliary Tasks for Multitask Learning](https://mlsanity.com/paper/2405.07769/) (2024)
- [From Insight to Exploit: Leveraging LLM Collaboration for Adaptive Adversarial Text Generation](https://mlsanity.com/paper/2511.03128/) (2025)

## Abstract (arXiv)

Many NLP tasks have benefited from transferring knowledge from contextualized word embeddings, however the picture of what type of knowledge is transferred is incomplete. This paper studies the types of linguistic phenomena accounted for by language models in the context of a Conversational Question Answering (CoQA) task. We identify the problematic areas for the finetuned RoBERTa, BERT and DistilBERT models through systematic error analysis - basic arithmetic (counting phrases), compositional semantics (negation and Semantic Role Labeling), and lexical semantics (surprisal and antonymy). When enhanced with the relevant linguistic knowledge through multitask learning, the models improve in performance. Ensembles of the enhanced models yield a boost between 2.2 and 2.7 points in F1 score overall, and up to 42.1 points in F1 on the hardest question classes. The results show differences in ability to represent compositional and lexical information between RoBERTa, BERT and DistilBERT.