# What Are the Odds? Language Models Are Capable of Probabilistic Reasoning

arXiv: https://arxiv.org/abs/2406.12830 · HTML: https://mlsanity.com/paper/2406.12830/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [Transforming Wearable Data into Personal Health Insights using Large Language Model Agents](https://mlsanity.com/paper/2406.06464/) (2024)

## Built on by (most cited first, 26 of 26)

- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization](https://mlsanity.com/paper/2505.15918/) (2025)
- [Fourier Head: Helping Large Language Models Learn Complex Probability Distributions](https://mlsanity.com/paper/2410.22269/) (2024)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Bayesian Social Deduction with Graph-Informed Language Models](https://mlsanity.com/paper/2506.17788/) (2025)
- [Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMs](https://mlsanity.com/paper/2509.10739/) (2025)
- [OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data](https://mlsanity.com/paper/2510.15096/) (2025)
- [Can Revealed Preferences Clarify LLM Alignment and Steering?](https://mlsanity.com/paper/2605.08556/) (2026)
- [Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces](https://mlsanity.com/paper/2609.37470/) (2026)
- [Improving the Distributional Alignment of LLMs using Supervision](https://mlsanity.com/paper/2507.00439/) (2025)
- [Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models](https://mlsanity.com/paper/2607.15277/) (2026)
- [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025)
- [NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction](https://mlsanity.com/paper/2511.09971/) (2025)
- [Humans and LLMs Diverge on Probabilistic Inferences](https://mlsanity.com/paper/2602.23546/) (2026)
- [High-Stakes Decisions with Language Models: Insights from Emergency Triage](https://mlsanity.com/paper/2608.01361/) (2026)
- [Enabling Fine-Grained Operating Points for Black-Box LLMs](https://mlsanity.com/paper/2510.17727/) (2025)
- [Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM](https://mlsanity.com/paper/2512.12868/) (2025)
- [Tutorial on Reasoning for IR & IR for Reasoning](https://mlsanity.com/paper/2602.03640/) (2026)
- [Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models](https://mlsanity.com/paper/2604.06201/) (2026)
- [Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins](https://mlsanity.com/paper/2607.09306/) (2026)
- [Bayesian Intelligence from the Outside](https://mlsanity.com/paper/2609.14724/) (2026)
- [Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow](https://mlsanity.com/paper/2610.00679/) (2026)
- [ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2610.11140/) (2026)
- [ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence](https://mlsanity.com/paper/2610.14381/) (2026)

## Abstract (arXiv)

Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle. An important but rarely evaluated form of reasoning is understanding probability distributions. In this paper, we focus on evaluating the probabilistic reasoning capabilities of LMs using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities. We evaluate three ways to provide context to LMs 1) anchoring examples from within a distribution or family of distributions, 2) real-world context, 3) summary statistics on which to base a Normal approximation. Models can make inferences about distributions, and can be further aided by the incorporation of real-world context, example shots and simplified assumptions, even if these assumptions are incorrect or misspecified. To conduct this work, we developed a comprehensive benchmark distribution dataset with associated question-answer pairs that we have released publicly.