# Compact Language, Complex Model Shifts: How and Where Ambiguity and Underspecification Affect LLMs

arXiv: https://arxiv.org/abs/2609.39572 · HTML: https://mlsanity.com/paper/2609.39572/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)
- [Aligning Language Models to Explicitly Handle Ambiguity](https://mlsanity.com/paper/2404.11972/) (2024)
- [Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords](https://mlsanity.com/paper/2109.11491/) (2021)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)

## Abstract (arXiv)

We analyze how lexical ambiguity and underspecification affect language model training. We create artificial homonyms and artificial hypernyms as pseudowords and analyze the generative performance of language models as they are trained with increasing amounts of these ambiguous or underspecified pseudoword types. We further analyze whether the models disambiguate ambiguous or underspecified statements and provide a first mechanistic account of how ambiguity and disambiguation are represented internally. Our main results show that both ambiguity and underspecification increase model performance in ways that scale with their influence on the language's type-token ratio. However, the accuracy of generating sequences containing ambiguous words or their synonyms decreases compared to other texts. We also show that internal representations of pseudowords reflect disambiguation of pseudo-homonyms, but underspecification of pseudo-hypernyms is maintained during the generative process.