# Spelling convention sensitivity in neural language models

arXiv: https://arxiv.org/abs/2303.03457 · HTML: https://mlsanity.com/paper/2303.03457/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset](https://mlsanity.com/paper/2007.01176/) (2020)
- [RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency](https://mlsanity.com/paper/1809.01329/) (2018)

## Built on by (most cited first, 1 of 1)

- [How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline](https://mlsanity.com/paper/2604.04204/) (2026)

## Abstract (arXiv)

We examine whether large neural language models, trained on very large collections of varied English text, learn the potentially long-distance dependency of British versus American spelling conventions, i.e., whether spelling is consistently one or the other within model-generated strings. In contrast to long-distance dependencies in non-surface underlying structure (e.g., syntax), spelling consistency is easier to measure both in LMs and the text corpora used to train them, which can provide additional insight into certain observed model behaviors. Using a set of probe words unique to either British or American English, we first establish that training corpora exhibit substantial (though not total) consistency. A large T5 language model does appear to internalize this consistency, though only with respect to observed lexical items (not nonce words with British/American spelling patterns). We further experiment with correcting for biases in the training data by fine-tuning T5 on synthetic data that has been debiased, and find that finetuned T5 remains only somewhat sensitive to spelling consistency. Further experiments show GPT2 to be similarly limited.