What Does a Sharing Question Add? Auditing LLM Survey Scores for Misinformation
Not on the map. Why?
Abstract (arXiv)
Evaluating misinformation requires distinguishing whether readers believe content from whether they would share it. Asking large language models (LLMs) both questions yields two scores, but does the sharing answer contribute information beyond the credibility answer? We audit eight model versions on 290 synthetic misinformation articles, using 1,256 paired survey responses with 317 participant identifiers as an external validity criterion. An initial reversal motivates the audit: every model's raw sharing score predicts mean human sharing less accurately than its credibility score. This ordering changes after offset correction, so it does not by itself diagnose missing information. We instead distinguish score reconstructability, persistence across elicitation formats, and incremental human validity. Credibility predicts 30.8-72.5% of model-sharing variation relative to a held-out constant baseline; remaining sharing differences correlate at 0.61-0.75 across question-order and separate-question conditions. Yet adding model sharing to human and model credibility yields only -0.25% to +0.69% error reduction with fixed regression, with all exploratory intervals crossing zero. Flexible prediction and format changes do not establish an improvement. Sharing answers therefore contain structured variation beyond the observed credibility score, without established incremental validity for human sharing in these data. The findings motivate validating the contribution of each elicited outcome, beyond inspecting score differences or agreement across prompts.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 16 references that are in the map. First posted on arXiv 8 Apr 2026.
Builds on
5 of 16 references.
References · 5
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
- 2023Questioning the Survey Responses of Large Language Modelscited by 74
- 2026Overstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibilitycited by 5
Built on by
Not cited yet.