# Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers

arXiv: https://arxiv.org/abs/2607.13304 · HTML: https://mlsanity.com/paper/2607.13304/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 22 references that are in the map.
- First posted on arXiv 14 Jul 2026.

## Builds on (most cited first, 3 of 3)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)

## Built on by (most cited first, 4 of 4)

- [The Language of the Question Selects the Market: Query Language and Exit IP as Separable Factors in Commercial Recommendations from a Generative Search Interface](https://mlsanity.com/paper/2608.30052/) (2026)
- [Who Gets Named: Citation Type Predicts Individual Naming by Grounded Language Models, and a Roster Instrument Captures 0.5% of It](https://mlsanity.com/paper/2607.23893/) (2026)
- [Ask a Language Model for Lottery Numbers: Concentration in Repeated Six-of-49 Outputs](https://mlsanity.com/paper/2610.00052/) (2026)
- [Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers](https://mlsanity.com/paper/2610.19608/) (2026)

## Abstract (arXiv)

Teams measuring whether large language models (LLMs) recommend a brand face a reproducibility problem: ask the same question twice and the answer moves. Practice resamples each prompt a few times (commonly five) and averages, treating within-prompt resampling as the source of the noise. But a measured brand score moves for at least four separable reasons: within-prompt resampling, prompt paraphrase, model identity, and query language. We specify a crossed random-effects (generalizability-theory) decomposition that partitions the total variance of a response-level brand outcome into these four sources, and embed the components in a decision-study allocation that returns how many repeats, paraphrases, models, and languages to buy for a target reliability. We apply it to a fully crossed corpus of 12,933 LLM responses on 20 Central and Eastern European brands, 8 languages, and 3 models (GPT-5.2 and Gemini 3 Flash in parametric mode, Perplexity in grounded retrieval), with a stability subset of 1,435 cells resampled about five times. The outcome is per-response multilingual sentiment polarity. Query language is the largest systematic facet (26.5% of the variance of one response) against 1.5% for brand identity (ICC 0.0146), so a single AI answer carries almost no brand-discriminating signal. Once a cell term isolates pure resampling, resampling is 34.8% of variance and the brand-in-context interaction 29.6%; brand-by-language is 8.6% (a bilingual penalty) while brand-by-model and brand-by-prompt are near zero. Per unit of query budget, adding languages and models reduces relative-error variance far more than adding repeats: a repeat past the fifth reduces it by only 0.0003. Brand-ranking reliability stays low, near 0.01 for a single answer and about 0.36 at the full crossed design, so reliability is bought by spreading across languages and models, not by repeating one prompt.