# How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models

arXiv: https://arxiv.org/abs/2402.05034 · HTML: https://mlsanity.com/paper/2402.05034/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 13 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://mlsanity.com/paper/2008.06460/) (2020)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models](https://mlsanity.com/paper/2101.09688/) (2021)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)

## Built on by (most cited first, 2 of 2)

- [Analyzing Semantic Change through Lexical Replacements](https://mlsanity.com/paper/2404.18570/) (2024)
- [Values That Are Explicitly Present in Fairy Tales: Comparing Samples from German, Italian and Portuguese Traditions](https://mlsanity.com/paper/2402.08318/) (2024)

## Abstract (arXiv)

In this paper, we explore the idea of analysing the historical bias of contextual language models based on BERT by measuring their adequacy with respect to Early Modern (EME) and Modern (ME) English. In our preliminary experiments, we perform fill-in-the-blank tests with 60 masked sentences (20 EME-specific, 20 ME-specific and 20 generic) and three different models (i.e., BERT Base, MacBERTh, English HLM). We then rate the model predictions according to a 5-point bipolar scale between the two language varieties and derive a weighted score to measure the adequacy of each model to EME and ME varieties of English.