mlsanityData to 14 Oct 2026

Always Tell Me The Odds: Fine-grained Conditional Probability Estimation

arXiv 2505.01595 ↗First posted 2 May 2025cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We present a state-of-the-art model for fine-grained probability estimation of propositions conditioned on context. Recent advances in large language models (LLMs) have significantly enhanced their reasoning capabilities, particularly on well-defined tasks with complete information. However, LLMs continue to struggle with making accurate and well-calibrated probabilistic predictions under uncertainty or partial information. While incorporating uncertainty into model predictions often boosts performance, obtaining reliable estimates of that uncertainty remains understudied. In particular, LLM probability estimates tend to be coarse and biased towards more frequent numbers. Through a combination of human and synthetic data creation and assessment, scaling to larger models, and better supervision, we propose a set of strong and precise probability estimation models. We conduct systematic evaluations across tasks that rely on conditional probability estimation and show that our approach consistently outperforms existing fine-tuned and prompting-based methods by a large margin.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 90 of its 135 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

90 of 135 references.

References · 90

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2023Let's Verify Step by Stepcited by 3,789
38 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2024A Survey on LLM-as-a-Judgecited by 1,423
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  13. 2019Measuring Calibration in Deep Learningcited by 446
  14. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  15. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  16. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  17. 2020Calibration of Pre-trained Transformerscited by 315
  18. 2019Verified Uncertainty Calibrationcited by 309
  19. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  20. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  21. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  22. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  23. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  24. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  25. 2021Explaining Answers with Entailment Treescited by 152
  26. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  27. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  28. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  29. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  30. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  31. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  32. 2021Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibrationcited by 79
  33. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  34. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  35. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  36. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  37. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  38. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaborationcited by 51

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMscited by 8
  2. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  3. 2026Unified Multimodal Uncertain Inferencecited by 3
  4. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 2
  5. 2026Scaling Participation in Modular AI Systemscited by 2
  6. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  7. 2026Humans and LLMs Diverge on Probabilistic Inferencescited by 1
  8. 2026FrankenReport: Early Exiting in Long-Form Generation Using Expected Value of Computationcited by 0
  9. 2026PRISM-BN: A Controlled Corpus and Benchmark for Text-to-Parameterized Bayesian Network Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking LLM Competence on Logical Inference over Probability Operators
  2. 2025Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution
  3. 2024Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning
  4. 2024Reasoning over Uncertain Text by Generative Large Language Models
  5. 2026Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
  6. 2025Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs
  7. 2026Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
  8. 2025TextBandit: Evaluating Probabilistic Reasoning in LLMs Through Language-Only Decision Tasks