What Makes a Good Query? Measuring the Impact of Human-Confusing Linguistic Features on LLM Performance
Not on the map. Why?
Abstract (arXiv)
Large Language Model (LLM) hallucinations are usually treated as defects of the model or its decoding strategy. Drawing on classical linguistics, we argue that a query's form can also shape a listener's (and model's) response. We operationalize this insight by constructing a 22-dimension query feature vector covering clause complexity, lexical rarity, and anaphora, negation, answerability, and intention grounding, all known to affect human comprehension. Using 369,837 real-world queries, we ask: Are there certain types of queries that make hallucination more likely? A large-scale analysis reveals a consistent "risk landscape": certain features such as deep clause nesting and underspecification align with higher hallucination propensity. In contrast, clear intention grounding and answerability align with lower hallucination rates. Others, including domain specificity, show mixed, dataset- and model-dependent effects. Thus, these findings establish an empirically observable query-feature representation correlated with hallucination risk, paving the way for guided query rewriting and future intervention studies.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 122 references that are in the map. First posted on arXiv 23 Feb 2026.
Builds on
41 of 122 references.
References · 41
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
29 more
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
- 2017Crowdsourcing Multiple Choice Science Questionscited by 742
- 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
- 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
- 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
- 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
- 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
- 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
- 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
- 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
- 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
- 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 66
- 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
- 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 55
- 2021Sparse Attention with Linear Unitscited by 41
- 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 40
- 2022Prompting Language Models for Linguistic Structurecited by 39
- 2018Anaphora and Coreference Resolution: A Reviewcited by 31
- 2024FlowMind: Automatic Workflow Generation with LLMscited by 30
- 2017Are you serious?: Rhetorical Questions and Sarcasm in Social Media Dialogcited by 7
- 2024Is There No Such Thing as a Bad Question? H4R: HalluciBot For Ratiocination, Rewriting, Ranking, and Routingcited by 6
- 2025MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answerscited by 2
Built on by
Not cited yet.