mlsanityData to 14 Oct 2026

Self-Verification Improves Few-Shot Clinical Information Extraction

arXiv 2306.00024 ↗First posted 30 May 2023cs.CL cs.LGcited by 52

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Extracting patient information from unstructured text is a critical task in health decision-support and clinical research. Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning, in contrast to supervised learning which requires much more costly human annotations. However, despite drastic advances in modern LLMs such as GPT-4, they still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs. This is made possible by the asymmetry between verification and generation, where the latter is often much easier than the former. Experimental results show that our method consistently improves accuracy for various LLMs in standard clinical information extraction tasks. Additionally, self-verification yields interpretations in the form of a short text span corresponding to each output, which makes it very efficient for human experts to audit the results, paving the way towards trustworthy extraction of clinical information in resource-constrained scenarios. To facilitate future research in this direction, we release our code and prompts.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 28 of its 43 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

28 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again16 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  10. 2019Attention is not not Explanationcited by 636
  11. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  12. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
15 more
  1. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  2. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  3. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  4. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  5. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  6. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  7. 2023Training Language Models with Language Feedback at Scalecited by 128
  8. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  9. 2018A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literaturecited by 79
  10. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  11. 2023Explaining black box text modules in natural language with language modelscited by 62
  12. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 54
  13. 2023Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Studycited by 49
  14. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  15. 2023GPT is becoming a Turing machine: Here are some ways to program itcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 29; 2025: 13; 2026: 3.

52 papers.

Sort:

By area:

  1. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  2. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  3. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  4. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  5. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  6. 2025Confidence Improves Self-Consistency in LLMscited by 113
  7. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  8. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 102
  9. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  10. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  11. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  12. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
38 more
  1. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  2. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
  3. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  4. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  5. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  6. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 23
  7. 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Frameworkcited by 19
  8. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 19
  9. 2024CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generationcited by 18
  10. 2023Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflowscited by 17
  11. 2024Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinementcited by 16
  12. 2025Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Factscited by 13
  13. 2023Sentiment Analysis through LLM Negotiationscited by 11
  14. 2025How Well Can Reasoning Models Identify and Recover from Unhelpful Thoughts?cited by 11
  15. 2025Code Like Humans: A Multi-Agent Solution for Medical Codingcited by 11
  16. 2024Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Fieldcited by 10
  17. 2024On-the-fly Definition Augmentation of LLMs for Biomedical NERcited by 9
  18. 2024GeneAgent: Self-verification Language Agent for Gene Set Knowledge Discovery using Domain Databasescited by 9
  19. 2025MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaborationcited by 9
  20. 2023Surpassing GPT-4 Medical Coding with a Two-Stage Approachcited by 6
  21. 2024Data Analysis in the Era of Generative AIcited by 6
  22. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  23. 2024Best Practices for Large Language Models in Radiologycited by 5
  24. 2024Learning to Check: Unleashing Potentials for Self-Correction in Large Language Modelscited by 4
  25. 2024Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summariescited by 4
  26. 2024Extracting Social Determinants of Health from Pediatric Patient Notes Using Large Language Models: Novel Corpus and Methodscited by 4
  27. 2024Entity Decomposition with Filtering: A Zero-Shot Clinical Named Entity Recognition Frameworkcited by 4
  28. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4
  29. 2025Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstructioncited by 3
  30. 2025CardioCoT: Hierarchical Reasoning for Multimodal Survival Analysiscited by 3
  31. 2025MedDCR: Learning to Design Agentic Workflows for Medical Codingcited by 3
  32. 2024An Empirical Study on Self-correcting Large Language Models for Data Science Code Generationcited by 2
  33. 2024CACER: Clinical Concept Annotations for Cancer Events and Relationscited by 2
  34. 2025CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Modelscited by 2
  35. 2026Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extractioncited by 2
  36. 2024Generative AI in Medicinecited by 1
  37. 2025Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulationscited by 1
  38. 2025A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Healthcited by 1

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings
  2. 2024LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction
  3. 2023Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events
  4. 2025Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning
  5. 2024A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences
  6. 2024Large Language Models Struggle in Token-Level Clinical Named Entity Recognition
  7. 2024PrecLLM: A Privacy-Preserving Framework for Efficient Clinical Annotation Extraction from Unstructured EHRs using Small-Scale LLMs
  8. 2025VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records