mlsanityData to 14 Oct 2026

LLMs Accelerate Annotation for Medical Information Extraction

arXiv 2312.02296 ↗First posted 4 Dec 2023cs.CL cs.AI cs.LGcited by 77

Not on the map. Why?

Abstract (arXiv)

The unstructured nature of clinical notes within electronic health records often conceals vital patient-related information, making it challenging to access or interpret. To uncover this hidden information, specialized Natural Language Processing (NLP) models are required. However, training these models necessitates large amounts of labeled data, a process that is both time-consuming and costly when relying solely on human experts for annotation. In this paper, we propose an approach that combines Large Language Models (LLMs) with human expertise to create an efficient method for generating ground truth labels for medical text annotation. By utilizing LLMs in conjunction with human annotators, we significantly reduce the human annotation burden, enabling the rapid creation of labeled datasets. We rigorously evaluate our method on a medical information extraction task, demonstrating that our approach not only substantially cuts down on human intervention but also maintains high accuracy. The results highlight the potential of using LLMs to improve the utilization of unstructured clinical data, allowing for the swift deployment of tailored NLP solutions in healthcare.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 13 of its 40 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

13 of 40 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  6. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  7. 2022Is GPT-3 a Good Data Annotator?cited by 208
  8. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  9. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  10. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  11. 2019A Survey on Recent Advances in Named Entity Recognition from Deep Learning modelscited by 117
  12. 2019Enhancing Clinical Concept Extraction with Contextual Embeddingscited by 48
1 more
  1. 2018Clinical Concept Extraction with Contextual Word Embeddingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 44; 2025: 21; 2026: 12.

77 papers.

Sort:

By area:

  1. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  2. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 43
  3. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 42
  4. 2024Large Language Models for Medicine: A Surveycited by 38
  5. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  6. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  7. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 34
  8. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  9. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  10. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 25
  11. 2024DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledgecited by 24
  12. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
38 more
  1. 2024Using LLMs for the Extraction and Normalization of Product Attribute Valuescited by 20
  2. 2024Natural Language Processing in the Patent Domain: A Surveycited by 20
  3. 2024Can Large Language Models abstract Medical Coded Language?cited by 20
  4. 2024From Text to Transformation: A Comprehensive Review of Large Language Models' Versatilitycited by 19
  5. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  6. 2024MBIAS: Mitigating Bias in Large Language Models While Retaining Contextcited by 15
  7. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  8. 2024RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplementscited by 11
  9. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Predictioncited by 10
  10. 2024Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?cited by 9
  11. 2024Fine-Tuning a Local LLaMA-3 Large Language Model for Automated Privacy-Preserving Physician Letter Generation in Radiation Oncologycited by 8
  12. 2024CMNEE: A Large-Scale Document-Level Event Extraction Dataset based on Open-Source Chinese Military Newscited by 6
  13. 2024DAC: Decomposed Automation Correction for Text-to-SQLcited by 6
  14. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  15. 2024Data Analysis in the Era of Generative AIcited by 6
  16. 2024Automated Clinical Data Extraction with Knowledge Conditioned LLMscited by 5
  17. 2024Identifying Reasons for Contraceptive Switching from Real-World Data Using Large Language Modelscited by 4
  18. 2024GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimizationcited by 4
  19. 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
  20. 2025EchoingECG: An Electrocardiogram Cross-Modal Model for Echocardiogram Taskscited by 4
  21. 2024Leveraging Large Language Models for Web Scrapingcited by 3
  22. 2024Exploring Large Language Models for Product Attribute Value Identificationcited by 3
  23. 2024CurateGPT: A flexible language-model assisted biocuration toolcited by 3
  24. 2025JobHop: A Large-Scale Dataset of Career Trajectoriescited by 3
  25. 2024PrivComp-KG : Leveraging Knowledge Graph and Large Language Models for Privacy Policy Compliance Verificationcited by 2
  26. 2024Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insightscited by 2
  27. 2025Self-Refinement Strategies for LLM-based Product Attribute Value Extractioncited by 2
  28. 2025X2CT-CLIP: Enable Multi-Abnormality Detection in Computed Tomography from Chest Radiography via Tri-Modal Contrastive Learningcited by 2
  29. 2025CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reportscited by 2
  30. 2025Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Studycited by 2
  31. 2024Scaling Technology Acceptance Analysis with Large Language Model (LLM) Annotation Systemscited by 1
  32. 2024SDoH-GPT: Using Large Language Models to Extract Social Determinants of Health (SDoH)cited by 1
  33. 2024SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularizationcited by 1
  34. 2024Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modelingcited by 1
  35. 2025From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Modelscited by 1
  36. 2025Automated Archival Descriptions with Federated Intelligence of LLMscited by 1
  37. 2025Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Datacited by 1
  38. 2025Refine Medical Diagnosis Using Generation Augmented Retrieval and Clinical Practice Guidelinescited by 1

50 most cited of 77 All 77 (JSON)