mlsanityData to 7 Oct 2026

What Are the Odds? Language Models Are Capable of Probabilistic Reasoning

arXiv 2406.12830 ↗First posted 18 Jun 2024cs.CLcited by 25

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle. An important but rarely evaluated form of reasoning is understanding probability distributions. In this paper, we focus on evaluating the probabilistic reasoning capabilities of LMs using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities. We evaluate three ways to provide context to LMs 1) anchoring examples from within a distribution or family of distributions, 2) real-world context, 3) summary statistics on which to base a Normal approximation. Models can make inferences about distributions, and can be further aided by the incorporation of real-world context, example shots and simplified assumptions, even if these assumptions are incorrect or misspecified. To conduct this work, we developed a comprehensive benchmark distribution dataset with associated question-answer pairs that we have released publicly.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 17 of its 38 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

17 of 38 references.

References · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  9. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  10. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  11. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  12. 2024The pitfalls of next-token predictioncited by 156
5 more
  1. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  2. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  3. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  4. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  5. 2024Transforming Wearable Data into Personal Health Insights using Large Language Model Agentscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 11.

25 papers.

Sort:

By area:

  1. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  2. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  3. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  4. 2025Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterizationcited by 15
  5. 2024Fourier Head: Helping Large Language Models Learn Complex Probability Distributionscited by 10
  6. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  7. 2025Bayesian Social Deduction with Graph-Informed Language Modelscited by 8
  8. 2025Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMscited by 8
  9. 2026Can Revealed Preferences Clarify LLM Alignment and Steering?cited by 4
  10. 2025OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Datacited by 3
  11. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  12. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
13 more
  1. 2025Bonsai: Interpretable Tree-Adaptive Grounded Reasoningcited by 1
  2. 2025NumPert: Numerical Perturbations to Probe Language Models for Veracity Predictioncited by 1
  3. 2026Humans and LLMs Diverge on Probabilistic Inferencescited by 1
  4. 2025Enabling Fine-Grained Operating Points for Black-Box LLMscited by 0
  5. 2025Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLMcited by 0
  6. 2026Tutorial on Reasoning for IR & IR for Reasoningcited by 0
  7. 2026Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Modelscited by 0
  8. 2026Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator winscited by 0
  9. 2026High-Stakes Decisions with Language Models: Insights from Emergency Triagecited by 0
  10. 2026Bayesian Intelligence from the Outsidecited by 0
  11. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 0
  12. 2026Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allowcited by 0
  13. 2026ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reasoning over Uncertain Text by Generative Large Language Models
  2. 2024Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models
  3. 2025Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs
  4. 2025On the Notion that Language Models Reason
  5. 2024Confidence in the Reasoning of Large Language Models
  6. 2026How reliable are LLMs when it comes to playing dice?
  7. 2025Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs
  8. 2024Refining Answer Distributions for Improved Large Language Model Reasoning