# Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!

arXiv: https://arxiv.org/abs/2402.12486 · HTML: https://mlsanity.com/paper/2402.12486/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [GRUEN for Evaluating Linguistic Quality of Generated Text](https://mlsanity.com/paper/2010.02498/) (2020)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [Underspecification in Scene Description-to-Depiction Tasks](https://mlsanity.com/paper/2210.05815/) (2022)
- [Zero and Few-shot Semantic Parsing with Ambiguous Inputs](https://mlsanity.com/paper/2306.00824/) (2023)
- [Are Language Models Worse than Humans at Following Prompts? It's Complicated](https://mlsanity.com/paper/2301.07085/) (2023)
- [When Language Models Fall in Love: Animacy Processing in Transformer Language Models](https://mlsanity.com/paper/2310.15004/) (2023)
- [Do You See What I Mean? Visual Resolution of Linguistic Ambiguities](https://mlsanity.com/paper/1603.08079/) (2016)

## Built on by (most cited first, 7 of 7)

- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs](https://mlsanity.com/paper/2412.13835/) (2024)
- [Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation](https://mlsanity.com/paper/2510.09390/) (2025)
- [NRR-Core: Non-Resolution Reasoning as a Computational Framework for Contextual Identity and Ambiguity Preservation](https://mlsanity.com/paper/2512.13478/) (2025)
- [Who is the richest club in the championship? Detecting and Rewriting Underspecified Questions Improve QA Performance](https://mlsanity.com/paper/2602.11938/) (2026)
- [Can LLMs Detect Ambiguous Plural Reference? An Analysis of Split-Antecedent and Mereological Reference](https://mlsanity.com/paper/2510.04581/) (2025)
- [NRR-Phi: A Typed External Text-to-State Interface and Update Contract for Inspectable Ambiguity-State Maintenance](https://mlsanity.com/paper/2601.19933/) (2026)

## Abstract (arXiv)

In everyday language use, speakers frequently utter and interpret sentences that are semantically underspecified, namely, whose content is insufficient to fully convey their message or interpret them univocally. For example, to interpret the underspecified sentence "Don't spend too much", which leaves implicit what (not) to spend, additional linguistic context or outside knowledge is needed. In this work, we propose a novel Dataset of semantically Underspecified Sentences grouped by Type (DUST) and use it to study whether pre-trained language models (LMs) correctly identify and interpret underspecified sentences. We find that newer LMs are reasonably able to identify underspecified sentences when explicitly prompted. However, interpreting them correctly is much harder for any LMs. Our experiments show that when interpreting underspecified sentences, LMs exhibit little uncertainty, contrary to what theoretical accounts of underspecification would predict. Overall, our study reveals limitations in current models' processing of sentence semantics and highlights the importance of using naturalistic data and communicative scenarios when evaluating LMs' language capabilities.