# BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models

arXiv: https://arxiv.org/abs/2404.12494 · HTML: https://mlsanity.com/paper/2404.12494/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks](https://mlsanity.com/paper/2305.17390/) (2023)
- [Break It Down: A Question Understanding Benchmark](https://mlsanity.com/paper/2001.11770/) (2020)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [Temporal Reasoning on Implicit Events from Distant Supervision](https://mlsanity.com/paper/2010.12753/) (2020)
- [Training Chain-of-Thought via Latent-Variable Inference](https://mlsanity.com/paper/2312.02179/) (2023)
- [COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences](https://mlsanity.com/paper/2106.00969/) (2021)
- [Temporal Common Sense Acquisition with Minimal Supervision](https://mlsanity.com/paper/2005.04304/) (2020)
- [Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning](https://mlsanity.com/paper/2210.12217/) (2022)
- [Uncertain Natural Language Inference](https://mlsanity.com/paper/1909.03042/) (2019)
- [Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?](https://mlsanity.com/paper/2311.09702/) (2023)
- [ThinkSum: Probabilistic reasoning over sets using large language models](https://mlsanity.com/paper/2210.01293/) (2022)
- [DeLLMa: Decision Making Under Uncertainty with Large Language Models](https://mlsanity.com/paper/2402.02392/) (2024)
- [Conceptual and Unbiased Reasoning in Language Models](https://mlsanity.com/paper/2404.00205/) (2024)
- [Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts](https://mlsanity.com/paper/2210.16865/) (2022)
- [Generic Temporal Reasoning with Differential Analysis and Explanation](https://mlsanity.com/paper/2212.10467/) (2022)

## Built on by (most cited first, 18 of 18)

- [Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization](https://mlsanity.com/paper/2505.15918/) (2025)
- [Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models](https://mlsanity.com/paper/2507.12547/) (2025)
- [Calibrating Verbalized Confidence with Self-Generated Distractors](https://mlsanity.com/paper/2509.25532/) (2025)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems](https://mlsanity.com/paper/2506.10060/) (2025)
- [Let's Think Var-by-Var: Large Language Models Enable Ad Hoc Probabilistic Reasoning](https://mlsanity.com/paper/2412.02081/) (2024)
- [OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data](https://mlsanity.com/paper/2510.15096/) (2025)
- [Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning](https://mlsanity.com/paper/2512.02914/) (2025)
- [DeFine: Decision-Making with Analogical Reasoning over Factor Profiles](https://mlsanity.com/paper/2410.01772/) (2024)
- [AdaptFuse: Training-Free Sequential Preference Inference via Externalized Bayesian Computation](https://mlsanity.com/paper/2604.03925/) (2026)
- [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025)
- [Interpretable Probability Estimation with LLMs via Shapley Reconstruction](https://mlsanity.com/paper/2601.09151/) (2026)
- [Probabilistic Programs of Thought](https://mlsanity.com/paper/2604.17290/) (2026)
- [CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning](https://mlsanity.com/paper/2512.18857/) (2025)
- [Tutorial on Reasoning for IR & IR for Reasoning](https://mlsanity.com/paper/2602.03640/) (2026)
- [Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks](https://mlsanity.com/paper/2603.11689/) (2026)
- [Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow](https://mlsanity.com/paper/2610.00679/) (2026)
- [ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2610.11140/) (2026)

## Abstract (arXiv)

Predictive models often need to work with incomplete information in real-world tasks. Consequently, they must provide reliable probability or confidence estimation, especially in large-scale decision-making and planning tasks. Current large language models (LLMs) are insufficient for accurate estimations, but they can generate relevant factors that may affect the probabilities, produce coarse-grained probabilities when the information is more complete, and help determine which factors are relevant to specific downstream contexts. In this paper, we make use of these capabilities of LLMs to provide a significantly more accurate probabilistic estimation. We propose BIRD, a novel probabilistic inference framework that aligns a Bayesian network with LLM abductions and then estimates more accurate probabilities in a deduction step. We show BIRD provides reliable probability estimations that are 30% better than those provided directly by LLM baselines. These estimates further contribute to better and more trustworthy decision making.