# Hey, wait a minute: on at-issue sensitivity in Language Models

arXiv: https://arxiv.org/abs/2510.12740 · HTML: https://mlsanity.com/paper/2510.12740/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 39 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it](https://mlsanity.com/paper/2205.03472/) (2022)
- [Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion](https://mlsanity.com/paper/2210.05905/) (2022)

## Built on by (most cited first, 2 of 2)

- [Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs](https://mlsanity.com/paper/2601.04435/) (2026)
- [Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models](https://mlsanity.com/paper/2610.13794/) (2026)

## Abstract (arXiv)

Evaluating the naturalness of dialogue in language models (LMs) is not trivial: notions of 'naturalness' vary, and scalable quantitative metrics remain limited. This study leverages the linguistic notion of 'at-issueness' to assess dialogue naturalness and introduces a new method: Divide, Generate, Recombine, and Compare (DGRC). DGRC (i) divides a dialogue as a prompt, (ii) generates continuations for subparts using LMs, (iii) recombines the dialogue and continuations, and (iv) compares the likelihoods of the recombined sequences. This approach mitigates bias in linguistic analyses of LMs and enables systematic testing of discourse-sensitive behavior. Applying DGRC, we find that LMs prefer to continue dialogue on at-issue content, with this effect enhanced in instruct-tuned models. They also reduce their at-issue preference when relevant cues (e.g., "Hey, wait a minute") are present. Although instruct-tuning does not further amplify this modulation, the pattern reflects a hallmark of successful dialogue dynamics.