# Abstraction not Memory: BERT and the English Article System

arXiv: https://arxiv.org/abs/2206.04184 · HTML: https://mlsanity.com/paper/2206.04184/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Retrospective Reader for Machine Reading Comprehension](https://mlsanity.com/paper/2001.09694/) (2020)
- [Correlating neural and symbolic representations of language](https://mlsanity.com/paper/1905.06401/) (2019)
- [CxGBERT: BERT meets Construction Grammar](https://mlsanity.com/paper/2011.04134/) (2020)
- [Data Weighted Training Strategies for Grammatical Error Correction](https://mlsanity.com/paper/2008.02976/) (2020)

## Built on by (most cited first, 1 of 1)

- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)

## Abstract (arXiv)

Article prediction is a task that has long defied accurate linguistic description. As such, this task is ideally suited to evaluate models on their ability to emulate native-speaker intuition. To this end, we compare the performance of native English speakers and pre-trained models on the task of article prediction set up as a three way choice (a/an, the, zero). Our experiments with BERT show that BERT outperforms humans on this task across all articles. In particular, BERT is far superior to humans at detecting the zero article, possibly because we insert them using rules that the deep neural model can easily pick up. More interestingly, we find that BERT tends to agree more with annotators than with the corpus when inter-annotator agreement is high but switches to agreeing more with the corpus as inter-annotator agreement drops. We contend that this alignment with annotators, despite being trained on the corpus, suggests that BERT is not memorising article use, but captures a high level generalisation of article use akin to human intuition.