# Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ

arXiv: https://arxiv.org/abs/2601.16508 · HTML: https://mlsanity.com/paper/2601.16508/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [Do Language Models Exhibit Human-like Structural Priming Effects?](https://mlsanity.com/paper/2406.04847/) (2024)
- [As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making](https://mlsanity.com/paper/2501.12868/) (2025)
- [When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents](https://mlsanity.com/paper/2512.02445/) (2025)
- [Early ChatGPT User Portrait through the Lens of Data](https://mlsanity.com/paper/2312.10078/) (2023)
- [Towards Harnessing Large Language Models for Comprehension of Conversational Grounding](https://mlsanity.com/paper/2406.01749/) (2024)

## Abstract (arXiv)

Single-prompt evaluations dominate current LLM benchmarking, yet they fail to capture the conversational dynamics where real-world harm occurs. In this study, we examined whether conversation length affects response veracity by evaluating LLM performance on the BoolQ dataset under varying length and scaffolding conditions. Our results across three distinct LLMs revealed model-specific vulnerabilities that are invisible under single-turn testing. The length-dependent and scaffold-specific effects we observed demonstrate a fundamental limitation of static evaluations, as deployment-relevant vulnerabilities could only be spotted in a multi-turn conversational setting.