mlsanityData to 14 Oct 2026

Do LLMs estimate uncertainty well in instruction-following?

arXiv 2410.14582 ↗First posted 18 Oct 2024cs.AI cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) could be valuable personal AI agents across various domains, provided they can precisely follow user instructions. However, recent studies have shown significant limitations in LLMs' instruction-following capabilities, raising concerns about their reliability in high-stakes applications. Accurately estimating LLMs' uncertainty in adhering to instructions is critical to mitigating deployment risks. We present, to our knowledge, the first systematic evaluation of the uncertainty estimation abilities of LLMs in the context of instruction-following. Our study identifies key challenges with existing instruction-following benchmarks, where multiple factors are entangled with uncertainty stems from instruction-following, complicating the isolation and comparison across methods and models. To address these issues, we introduce a controlled evaluation setup with two benchmark versions of data, enabling a comprehensive comparison of uncertainty estimation methods under various conditions. Our findings show that existing uncertainty methods struggle, particularly when models make subtle errors in instruction following. While internal model states provide some improvement, they remain inadequate in more complex scenarios. The insights from our controlled evaluation setups provide a crucial understanding of LLMs' limitations and potential for uncertainty estimation in instruction-following tasks, paving the way for more trustworthy AI agents.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

30 of 35 references.

References · 30

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Mistral 7Bcited by 3,164
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
18 more
  1. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  3. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  4. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  5. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  6. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  7. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  8. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 91
  9. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 80
  10. 2024Towards Conversational Diagnostic AIcited by 79
  11. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 76
  12. 2023Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMscited by 62
  13. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 47
  14. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 35
  15. 2024Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Modelscited by 27
  16. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  17. 2024RefuteBench: Evaluating Refuting Instruction-Following for Large Language Modelscited by 11
  18. 2024Evalverse: Unified and Accessible Library for Large Language Model Evaluationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 14.

26 papers.

Sort:

By area:

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  2. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  3. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  4. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  5. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 14
  6. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  7. 2026Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agentscited by 8
  8. 2025LookAlike: Consistent Distractor Generation in Math MCQscited by 7
  9. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 7
  10. 2025Uncertainty Under the Curve: A Sequence-Level Entropy Area Metric for Reasoning LLMcited by 5
  11. 2026Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QAcited by 3
  12. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
14 more
  1. 2026Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agentscited by 2
  2. 2025I'm Sorry Dave: How the old world of personnel security can inform the new world of AI insider riskcited by 1
  3. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  4. 2026Self-Calibrating Language Models via Test-Time Discriminative Distillationcited by 1
  5. 2026From Sequence to Structure: Relational Uncertainty Propagation for LLM Agentscited by 1
  6. 2025Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMscited by 0
  7. 2025A Context-Aware Dual-Metric Framework for Confidence Estimation in Large Language Modelscited by 0
  8. 2026CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanationcited by 0
  9. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  10. 2026LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Datacited by 0
  11. 2026The Severance Problem: LLMs are Unaware of the Person Beyond the Promptcited by 0
  12. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  13. 2026Pinocchio: Fast Uncertainty Estimates for Black-Box Language Modelscited by 0
  14. 2026Stochastic Semantic Evidence Graphs: Uncertainty Propagation and Governance for Agentic AIcited by 0