Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) like ChatGPT have demonstrated amazing capabilities in comprehending user intents and generate reasonable and useful responses. Beside their ability to chat, their capabilities in various natural language processing (NLP) tasks are of interest to the research community. In this paper, we focus on assessing the overall ability of ChatGPT in 4 different medical information extraction (MedIE) tasks across 6 benchmark datasets. We present the systematically analysis by measuring ChatGPT's performance, explainability, confidence, faithfulness, and uncertainty. Our experiments reveal that: (a) ChatGPT's performance scores on MedIE tasks fall behind those of the fine-tuned baseline models. (b) ChatGPT can provide high-quality explanations for its decisions, however, ChatGPT is over-confident in its predcitions. (c) ChatGPT demonstrates a high level of faithfulness to the original text in the majority of cases. (d) The uncertainty in generation causes uncertainty in information extraction results, thus may hinder its applications in MedIE tasks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 85 of its 139 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
85 of 139 references.
References · 85
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2020Learning to summarize from human feedbackcited by 2,562
- 2023A Survey of Large Language Modelscited by 2,537
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
- 2023PaLM 2 Technical Reportcited by 1,154
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
- 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
- 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
- 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
- 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
- 2023Benchmarking Large Language Models for News Summarizationcited by 444
- 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
- 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
- 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
- 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
- 2023Mathematical Capabilities of ChatGPTcited by 263
- 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
- 2022Unified Structure Generation for Universal Information Extractioncited by 220
- 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
- 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
- 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
- 2021A Unified Generative Framework for Various NER Subtaskscited by 141
- 2023Unified Demonstration Retriever for In-Context Learningcited by 141
- 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
- 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
- 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99
- 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
- 2022ChatGPT: The End of Online Exam Integrity?cited by 88
- 2023ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Textcited by 84
50 most cited of 85 All 85 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.