mlsanityData to 14 Oct 2026

Incoherent Probability Judgments in Large Language Models

arXiv 2401.16646 ↗First posted 30 Jan 2024cs.CL cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

Autoregressive Large Language Models (LLMs) trained for next-word prediction have demonstrated remarkable proficiency at producing coherent text. But are they equally adept at forming coherent probability judgments? We use probabilistic identities and repeated judgments to assess the coherence of probability judgments made by LLMs. Our results show that the judgments produced by these models are often incoherent, displaying human-like systematic deviations from the rules of probability theory. Moreover, when prompted to judge the same event, the mean-variance relationship of probability judgments produced by LLMs shows an inverted-U-shaped like that seen in humans. We propose that these deviations from rationality can be explained by linking autoregressive LLMs to implicit Bayesian inference and drawing parallels with the Bayesian Sampler model of human probability judgments.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 12 of its 30 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

12 of 30 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2022Using cognitive psychology to understand GPT-3cited by 398
  7. 2019Verified Uncertainty Calibrationcited by 309
  8. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  9. 2023Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimationcited by 39
  10. 2023Bayes in the age of intelligent machinescited by 22
  11. 2018BRUNO: A Deep Recurrent Model for Exchangeable Datacited by 11
  12. 2023Deep de Finetti: Recovering Topic Distributions from Large Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 15.

30 papers.

Sort:

By area:

  1. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 109
  2. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  3. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 28
  4. 2025Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systemscited by 14
  5. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  6. 2024Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choicecited by 12
  7. 2025Levels of Analysis for Large Language Modelscited by 10
  8. 2025Towards Strategic Persuasion with Language Modelscited by 10
  9. 2026When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMscited by 10
  10. 2025Are Large Language Models Sensitive to the Motives Behind Communication?cited by 8
  11. 2025Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representationscited by 6
  12. 2026Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?cited by 6
18 more
  1. 2024Let's Think Var-by-Var: Large Language Models Enable Ad Hoc Probabilistic Reasoningcited by 5
  2. 2025Identifying and Mitigating the Influence of the Prior Distribution in Large Language Modelscited by 4
  3. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  4. 2025LLM Agents Display Human Biases but Exhibit Distinct Learning Patternscited by 3
  5. 2025Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Needcited by 3
  6. 2026Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theoremscited by 3
  7. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 3
  8. 2026Simulated Annealing Enhances Theory-of-Mind Reasoning in Autoregressive Language Modelscited by 2
  9. 2026Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Gamescited by 2
  10. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  11. 2026Do LLMs Share Human-Like Biases? Causal Reasoning Under Prior Knowledge, Irrelevant Context, and Varying Compute Budgetscited by 1
  12. 2026Beliefs and Behavior in Language Modelscited by 1
  13. 2026Eliciting Trustworthiness Priors of Large Language Models via Economic Gamescited by 0
  14. 2026Semantic Bayesian World Modelscited by 0
  15. 2026Bayesian Intelligence from the Outsidecited by 0
  16. 2026CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognitioncited by 0
  17. 2026Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanismcited by 0
  18. 2026Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allowcited by 0