mlsanityData to 7 Oct 2026

Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering

arXiv 2510.12925 ↗First posted 14 Oct 2025cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) should answer factual questions truthfully, grounded in objective knowledge, regardless of user context such as self-disclosed personal information, or system personalization. In this paper, we present the first systematic evaluation of LLM robustness to inquiry personas, i.e. user profiles that convey attributes like identity, expertise, or belief. While prior work has primarily focused on adversarial inputs or distractors for robustness testing, we evaluate plausible, human-centered inquiry persona cues that users disclose in real-world interactions. We find that such cues can meaningfully alter QA accuracy and trigger failure modes such as refusals, hallucinated limitations, and role confusion. These effects highlight how model sensitivity to user framing can compromise factual reliability, and position inquiry persona testing as an effective tool for robustness evaluation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 69 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

33 of 69 references.

References · 33

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
21 more
  1. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  2. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  3. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  4. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  5. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  6. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  7. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  8. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  9. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  10. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  11. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  12. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  13. 2022Multi-hop Question Answeringcited by 64
  14. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
  15. 2023CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Modelscited by 34
  16. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 33
  17. 2024Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humanscited by 31
  18. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  19. 2021Sorting through the noise: Testing robustness of information processing in pre-trained language modelscited by 27
  20. 2025More of the Same: Persistent Representational Harms Under Increased Representationcited by 6
  21. 2025Emoti-Attack: Zero-Perturbation Adversarial Attacks on NLP Systems via Emoji Sequencescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026PersonaDual: Balancing Personalization and Objectivity via Adaptive Reasoningcited by 0
  2. 2026PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AIcited by 0
  3. 2026StabilityBench: Benchmarking Instability in LLMscited by 0
  4. 2026Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describescited by 0