How reliable are LLMs when it comes to playing dice?
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems. We constructed two datasets, respectively a set of standard exercises and a set of counterintuitive exercises, designed to trigger heuristic reasoning, and evaluated 8 state-of-the-art models, each tested with and without Chain-of-Thought prompting. Models achieve an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones. We further provide empirical evidence of token bias: performance drops by over 20% when canonical formulations are replaced by disguised variants. Embedding misleading suggestions in the prompt reduces performance by up to 34%, with no model proving immune. Taken together, the reported findings suggest that current LLMs are not yet genuine probabilistic reasoners, despite their success in advanced mathematical problems.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 9 of its 20 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
9 of 20 references.
References · 9
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2023Bias and Fairness in Large Language Models: A Surveycited by 626
- 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
- 2022Thinking Fast and Slow in Large Language Modelscited by 136
- 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
- 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
- 2025Mathematical exploration and discovery at scalecited by 66
- 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
- 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models
- 2025Exploring LLM Reasoning Through Controlled Prompt Variations
- 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
- 2024Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models
- 2025Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMs
- 2025TextBandit: Evaluating Probabilistic Reasoning in LLMs Through Language-Only Decision Tasks
- 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
- 2025CER: Confidence Enhanced Reasoning in LLMs