# Signal or Spurious Cue? A Randomized Audit of Survey-Country Metadata in LLM Social Inference

arXiv: https://arxiv.org/abs/2608.06085 · HTML: https://mlsanity.com/paper/2608.06085/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 10 of 10)

- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting](https://mlsanity.com/paper/2406.11661/) (2024)
- [Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning](https://mlsanity.com/paper/2311.08385/) (2023)
- [Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment](https://mlsanity.com/paper/2604.12851/) (2026)
- [On the Credibility of Evaluating LLMs using Survey Questions](https://mlsanity.com/paper/2602.04033/) (2026)
- [Personalization, Personas, and Forecasting in Value Alignment](https://mlsanity.com/paper/2607.24782/) (2026)

## Abstract (arXiv)

Survey-country metadata can improve an LLM's forecast of an individual response when informative, yet the same cue may redirect the forecast when assigned at random. A within-record audit tests whether disclosing a random label's uniform, record-independent origin reduces its country-directed uptake, and whether verified survey country lowers held-out Brier loss. Independent population anchors and recorded human answers measure direction and consequence across five fixed API models, six countries, and seven development-selected targets. In the primary post-review 72-record panel, opaque and disclosed-random labels each produced country-direction shifts of 0.214. Paired attenuation was 0.0003 (95% CI [-0.0157, 0.0166]). Verified country reduced Brier loss by 0.040 (95% CI [0.024, 0.056]), while random-label regret included zero. A non-overlapping mixed-coverage consistency panel retained positive disclosed-random movement and verified utility, while attenuation remained uncertain. On the selected targets, verified metadata was useful in both panels, but disclosure did not reliably attenuate random-label uptake. PROV-FORECAST contains 14,400 paired item-level probability distributions from the corrected panel.