mlsanityData to 7 Oct 2026

Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs

arXiv 2310.11689 ↗First posted 18 Oct 2023cs.CL cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently shown great advances in a variety of tasks, including natural language understanding and generation. However, their use in high-stakes decision-making scenarios is still limited due to the potential for errors. Selective prediction is a technique that can be used to improve the reliability of the LLMs by allowing them to abstain from making predictions when they are unsure of the answer. In this work, we propose a novel framework for adaptation with self-evaluation to improve the selective prediction performance of LLMs. Our framework is based on the idea of using parameter-efficient tuning to adapt the LLM to the specific task at hand while improving its ability to perform self-evaluation. We evaluate our method on a variety of question-answering (QA) datasets and show that it outperforms state-of-the-art selective prediction methods. For example, on the CoQA benchmark, our method improves the AUACC from 91.23% to 92.63% and improves the AUROC from 74.61% to 80.25%.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

22 of 34 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
10 more
  1. 2017Selective Classification for Deep Neural Networkscited by 799
  2. 2021GPT Understands, Toocited by 715
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2018CoQA: A Conversational Question Answering Challengecited by 674
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  6. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  7. 2022Prompting GPT-3 To Be Reliablecited by 258
  8. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  9. 2023Planning with Large Language Models for Code Generationcited by 179
  10. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 17; 2026: 11.

49 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  3. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 79
  4. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scalingcited by 54
  5. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  6. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 44
  7. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
  8. 2024A Survey on the Honesty of Large Language Modelscited by 21
  9. 2025Eliciting Language Model Behaviors with Investigator Agentscited by 21
  10. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  11. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  12. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
37 more
  1. 2024Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answerscited by 18
  2. 2024Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Modelscited by 16
  3. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  4. 2024LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statementscited by 12
  5. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 12
  6. 2025The Confidence Paradox: Can LLM Know When It's Wrongcited by 11
  7. 2026Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)cited by 11
  8. 2024Attribute or Abstain: Large Language Models as Long Document Assistantscited by 10
  9. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 10
  10. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9
  11. 2024Overview of the EHRSQL 2024 Shared Task on Reliable Text-to-SQL Modeling on Electronic Health Recordscited by 9
  12. 2025Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundariescited by 7
  13. 2025SUGAR: Leveraging Contextual Confidence for Smarter Retrievalcited by 6
  14. 2025The Consistency Hypothesis in Uncertainty Quantification for Large Language Modelscited by 6
  15. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  16. 2024Efficiently Deploying LLMs with Controlled Riskcited by 5
  17. 2025Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Promptscited by 5
  18. 2025Variational Visual Question Answering for Uncertainty-Aware Selective Predictioncited by 5
  19. 2026On Calibration of Large Language Models: From Response To Capabilitycited by 5
  20. 2025Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classificationcited by 4
  21. 2025Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Processcited by 4
  22. 2024Recursive Chain-of-Feedback Prevents Performance Degradation from Redundant Promptingcited by 3
  23. 2024Topic Classification of Case Law Using a Large Language Model and a New Taxonomy for UK Law: AI Insights into Summary Judgmentcited by 3
  24. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Modelscited by 3
  25. 2024Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questionscited by 2
  26. 2025High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuningcited by 2
  27. 2024The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Casescited by 1
  28. 2025PIE: Performance Interval Estimation for Free-Form Generation Taskscited by 1
  29. 2025Reliable Inference in Edge-Cloud Model Cascades via Conformal Alignmentcited by 1
  30. 2026SIEVES: Selective Prediction Generalizes through Visual Evidence Scoringcited by 1
  31. 2026Aligning Language Models with Selective Predictioncited by 1
  32. 2025Improving VQA Reliability: A Dual-Assessment Approach with Self-Reflection and Cross-Model Verificationcited by 0
  33. 2026End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filteringcited by 0
  34. 2026Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classificationcited by 0
  35. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  36. 2026Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialoguecited by 0
  37. 2026Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inferencecited by 0