mlsanityData to 7 Oct 2026

Uncertainty Quantification for In-Context Learning of Large Language Models

arXiv 2402.10189 ↗First posted 15 Feb 2024cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) and revolutionized various fields by providing a few task-relevant demonstrations in the prompt. However, trustworthy issues with LLM's response, such as hallucination, have also been actively discussed. Existing works have been devoted to quantifying the uncertainty in LLM's response, but they often overlook the complex nature of LLMs and the uniqueness of in-context learning. In this work, we delve into the predictive uncertainty of LLMs associated with in-context learning, highlighting that such uncertainties may stem from both the provided demonstrations (aleatoric uncertainty) and ambiguities tied to the model's configurations (epistemic uncertainty). We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. Extensive experiments are conducted to demonstrate the effectiveness of the decomposition. The code and data are available at: https://github.com/lingchen0331/UQ_ICL.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

24 of 39 references.

References · 24

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2018Neural Network Acceptability Judgmentscited by 952
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  10. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  11. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  12. 2020Calibration of Pre-trained Transformerscited by 307
12 more
  1. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  2. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  3. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  4. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  5. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  6. 2020Uncertainty Aware Semi-Supervised Learning on Graph Datacited by 95
  7. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
  8. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  9. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 35
  10. 2022Source Localization of Graph Diffusion via Variational Autoencoders for Graph Inverse Problemscited by 18
  11. 2017Uncertainty Decomposition in Bayesian Neural Networks with Latent Variablescited by 9
  12. 2023Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertaintycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 19.

45 papers.

Sort:

By area:

  1. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
  2. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  3. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solvingcited by 31
  4. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  5. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  6. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 16
  7. 2025Language Model Uncertainty Quantification with Attention Chaincited by 13
  8. 2025Uncertainty Quantification and Decomposition for LLM-based Recommendationcited by 12
  9. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  10. 2025MAP: Multi-user Personalization with Collaborative LLM-powered Agentscited by 7
  11. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  12. 2024LLM Confidence Evaluation Measures in Zero-Shot CSS Classificationcited by 6
33 more
  1. 2026Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Modelcited by 6
  2. 2025Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?cited by 5
  3. 2026UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Modelscited by 5
  4. 2025Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selectioncited by 4
  5. 2025The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Modelscited by 4
  6. 2026Uncertainty Quantification for Flow-Based Vision-Language-Action Modelscited by 4
  7. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  8. 2025INSIGHT: INference-time Sequence Introspection for Generating Help Triggers in Vision-Language-Action Modelscited by 3
  9. 2026Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecastingcited by 3
  10. 2025From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learningcited by 2
  11. 2025Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulationcited by 2
  12. 2025ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantificationcited by 2
  13. 2026A Geometric Analysis of Small-sized Language Model Hallucinationscited by 2
  14. 2024LaMsS: When Large Language Models Meet Self-Skepticismcited by 1
  15. 2026Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrievalcited by 1
  16. 2026Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learningcited by 1
  17. 2026AGSC: Adaptive Granularity and Semantic Clustering for Uncertainty Quantification in Long-text Generationcited by 1
  18. 2026Diagnosing LLM Arbitration Behavior over Pre-evidence Epistemic States in RAG-based Fact-Checkingcited by 1
  19. 2025Performance Gains of LLMs With Humans in a World of LLMs Versus Humanscited by 0
  20. 2025Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMscited by 0
  21. 2025Stable LLM Ensemble: Interaction between Example Representativeness and Diversitycited by 0
  22. 2025Aligning Probabilistic Beliefs under Informative Missingness: LLM Steerability in Clinical Reasoningcited by 0
  23. 2025Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Errorcited by 0
  24. 2026Directional Concentration Uncertainty: A representational approach to uncertainty quantification for generative modelscited by 0
  25. 2026Visioning Human-Agentic AI Teaming: Continuity, Tension, and Future Researchcited by 0
  26. 2026A Formal Framework for Uncertainty Analysis of Text Generation with Large Language Modelscited by 0
  27. 2026CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systemscited by 0
  28. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0
  29. 2026CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptationcited by 0
  30. 2026Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuningcited by 0
  31. 2026Confidence Composition for Multiagent Language Model Systemscited by 0
  32. 2026Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidencecited by 0
  33. 2026Clarify the User or Verify the World? Uncertainty Routing for Proactive Agentscited by 0