How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversations
Not on the map. Why?
Abstract (arXiv)
Users articulate the same advice-seeking request in different ways: some specify detailed constraints, others gesture at a vague need. Prior work treats this variation as noise to be averaged away; we instead treat it as a stable, measurable structure in the input distribution. We ask whether articulation (how people ask) forms latent dimensions separable from topic (what they ask about), and whether it is associated with how language models respond. We extract interpretable features from 16,447 advice-seeking prompts pooled from public chat corpora (WildChat, LMSYS, and ShareChat) and recover a small set of latent articulation factors that replicate across train/test splits and across corpora. Because this structure is largely separable from topic, the populations it defines cut across topics and stay invisible to topic- or task-based evaluation. The factors define a handful of recurring articulation styles, one of which stands out: a long-form but information-poor style, roughly one in six prompts in the largest corpus, where models return shorter, vaguer answers and do not ask for clarification even though under-specification is exactly the condition that warrants it. The contrast holds within every topic group and length quintile, and is not under-specification alone -- a second, equally under-specified style does draw clarifying questions. Two independent human annotators reproduce this contrast. We argue that benchmarks should stratify on articulation, and we offer the extracted structure as a measurement instrument for doing so.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 30 Aug 2026.
Builds on
20 of 47 references.
References · 20
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
- 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 139
- 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 115
8 more
- 2023Eliciting Human Preferences with Language Modelscited by 92
- 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
- 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
- 2023Grounding Gaps in Language Model Generationscited by 47
- 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
- 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 29
- 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
- 2024Native Design Bias: Studying the Impact of English Nativeness on Language Model Performancecited by 5
Built on by
Not cited yet.