Signal or Spurious Cue? A Randomized Audit of Survey-Country Metadata in LLM Social Inference
Not on the map. Why?
Abstract (arXiv)
Survey-country metadata can improve an LLM's forecast of an individual response when informative, yet the same cue may redirect the forecast when assigned at random. A within-record audit tests whether disclosing a random label's uniform, record-independent origin reduces its country-directed uptake, and whether verified survey country lowers held-out Brier loss. Independent population anchors and recorded human answers measure direction and consequence across five fixed API models, six countries, and seven development-selected targets. In the primary post-review 72-record panel, opaque and disclosed-random labels each produced country-direction shifts of 0.214. Paired attenuation was 0.0003 (95% CI [-0.0157, 0.0166]). Verified country reduced Brier loss by 0.040 (95% CI [0.024, 0.056]), while random-label regret included zero. A non-overlapping mixed-coverage consistency panel retained positive disclosed-random movement and verified utility, while attenuation remained uncertain. On the selected targets, verified metadata was useful in both panels, but disclosure did not reliably attenuate random-label uptake. PROV-FORECAST contains 14,400 paired item-level probability distributions from the corrected panel.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 18 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
10 of 18 references.
References · 10
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2017Counterfactual Fairnesscited by 804
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
- 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
- 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
- 2023Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoningcited by 19
- 2026Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignmentcited by 5
- 2026On the Credibility of Evaluating LLMs using Survey Questionscited by 3
- 2026Personalization, Personas, and Forecasting in Value Alignmentcited by 1
Built on by
Not cited yet.