mlsanityData to 14 Oct 2026

Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models

arXiv 2505.11731 ↗First posted 16 May 2025cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) often exhibit misalignment between the quality of their generated responses and the confidence estimates they assign to them. Bayesian treatments, such as marginalizing over a reliable weight posterior or over the space of reasoning traces, provide an effective remedy, but incur substantial computational overhead due to repeated sampling at test time. To enable accurate uncertainty estimation in a single forward pass, we propose a novel distributional distillation framework (Dist2ill) that trains an LLM to produce multiple diverse reasoning paths within one inference pass, while using a lightweight parametric module to approximate empirical confidence scores derived from the sampling distribution. Extensive experiments demonstrate that Dist2ill preserves reasoning diversity and achieves state-of-the-art uncertainty estimation, substantially improving Expected Calibration Error (ECE) and Negative Log-Likelihood (NLL), while remaining computationally efficient.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 56 of its 83 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

56 of 83 references.

References · 56

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  13. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  14. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  15. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  16. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  17. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  18. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  19. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  20. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  21. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  22. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  23. 2025Confidence Improves Self-Consistency in LLMscited by 113
  24. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  25. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
  26. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  27. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  28. 2024On Verbalized Confidence Scores for LLMscited by 83
  29. 2023LoRA ensembles for large language model fine-tuningcited by 67
  30. 2023Selectively Answering Ambiguous Questionscited by 66
  31. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  32. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 63
  33. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 55
  34. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  35. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  36. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  37. 2015Bayesian Dark Knowledgecited by 45
  38. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 37

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistencycited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  2. 2026Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
  3. 2025Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMs
  4. 2026Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
  5. 2026Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation
  6. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  7. 2026Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
  8. 2026Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation