# The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations

arXiv: https://arxiv.org/abs/2609.04047 · HTML: https://mlsanity.com/paper/2609.04047/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 69 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Non-Determinism of "Deterministic" LLM Settings](https://mlsanity.com/paper/2408.04667/) (2024)
- [The Political Preferences of LLMs](https://mlsanity.com/paper/2402.01789/) (2024)
- [Fairness in Information Access Systems](https://mlsanity.com/paper/2105.05779/) (2021)
- [LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows](https://mlsanity.com/paper/2511.07585/) (2025)
- [Who Owns the AI Recommendation? A Multi-Industry Empirical Map of Brand Category Ownership Across Large Language Models](https://mlsanity.com/paper/2606.23057/) (2026)
- [The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages](https://mlsanity.com/paper/2606.23165/) (2026)
- [How Large Language Models Source Brand Reputation Across Languages and Markets](https://mlsanity.com/paper/2606.25787/) (2026)

## Built on by (most cited first, 2 of 2)

- [Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit](https://mlsanity.com/paper/2609.05059/) (2026)
- [Ask a Language Model for Lottery Numbers: Concentration in Repeated Six-of-49 Outputs](https://mlsanity.com/paper/2610.00052/) (2026)

## Abstract (arXiv)

Background: Researchers increasingly use repeated identical prompts to audit stochastic variation in large language model (LLM) brand recommendations, yet no standardized protocol exists for setting iteration counts, selecting stability metrics, or establishing reliability thresholds. Objective: We formalize the Dice Roll Method as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling. Methods: Total response variance is decomposed into sampling, prompt-phrasing, run-to-run, and model-version components. The stack: a negative-binomial mixed model with iterations as repeated measures; Cliff's delta as the distribution-free effect size; dependence-preserving bootstrap; simulation-based power; a generalizability-theory decomposition; drift diagnostics on pinned snapshots. We reanalyse five brand-recommendation auditing studies: approximately 190,000 observations, 270+ brands, 6 languages, iteration counts 5 to 40. Results: The recomputed D-study supports no fixed tiers. Across 250 distinct questions (S5), G is 0.96 at five runs and 1 to 2 runs reach 0.80, also within each model. For five versions of one question (S1 Valentine's subset, Gemini) it is 0.27 (95% CI 0.00 to 0.52), which fixes no run count. Runs follow from a pilot: n = 4 x residual / cell variance. Simulated single-cell power for a large Cliff's delta is 0.15 to 0.19 at five runs and 0.8 at 15 to 25. The four metric families (count, set, embedding, PASOR) are complementary. A pre-registered external validation on three independent corpora reproduces the D-study prediction in 37 of 39 cells with no failures; fixed tiers do not transfer. Conclusion: The protocol gives repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation.