mlsanityData to 14 Oct 2026

Large Language Models Struggle in Token-Level Clinical Named Entity Recognition

arXiv 2407.00731 ↗First posted 30 Jun 2024cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized various sectors, including healthcare where they are employed in diverse applications. Their utility is particularly significant in the context of rare diseases, where data scarcity, complexity, and specificity pose considerable challenges. In the clinical domain, Named Entity Recognition (NER) stands out as an essential task and it plays a crucial role in extracting relevant information from clinical texts. Despite the promise of LLMs, current research mostly concentrates on document-level NER, identifying entities in a more general context across entire documents, without extracting their precise location. Additionally, efforts have been directed towards adapting ChatGPT for token-level NER. However, there is a significant research gap when it comes to employing token-level NER for clinical texts, especially with the use of local open-source LLMs. This study aims to bridge this gap by investigating the effectiveness of both proprietary and local LLMs in token-level clinical NER. Essentially, we delve into the capabilities of these models through a series of experiments involving zero-shot prompting, few-shot prompting, retrieval-augmented generation (RAG), and instruction-fine-tuning. Our exploration reveals the inherent challenges LLMs face in token-level NER, particularly in the context of rare diseases, and suggests possible improvements for their application in healthcare. This research contributes to narrowing a significant gap in healthcare informatics and offers insights that could lead to a more refined application of LLMs in the healthcare sector.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

21 of 37 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Mistral 7Bcited by 3,164
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  9. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  10. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  11. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  12. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 72
9 more
  1. 2022BERN2: an advanced neural biomedical named entity recognition and normalization toolcited by 42
  2. 2023Label Supervised LLaMA Finetuningcited by 20
  3. 2023Identifying and Extracting Rare Disease Phenotypes with Large Language Modelscited by 18
  4. 2023Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPTcited by 18
  5. 2023Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processingcited by 16
  6. 2021The RareDis corpus: a corpus annotated with rare diseases, their signs and symptomscited by 11
  7. 2023AE-GPT: Using Large Language Models to Extract Adverse Events from Surveillance Reports-A Use Case with Influenza Vaccine Adverse Eventscited by 8
  8. 2021Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from textscited by 7
  9. 2024Improving Entity Recognition Using Ensembles of Deep Learning and Fine-tuned Large Language Models: A Case Study on Adverse Event Extraction from Multiple Sourcescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Benchmarking GPT-5 for biomedical natural language processingcited by 5
  2. 2025LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognitioncited by 1
  3. 2026FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Textscited by 1
  4. 2025Leveraging Large Language Models for Rare Disease Named Entity Recognitioncited by 0
  5. 2025Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Datasetcited by 0
  6. 2025Automated Identification of Incidentalomas Requiring Follow-Up: A Multi-Anatomy Evaluation of LLM-Based and Supervised Approachescited by 0
  7. 2026Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs in Biomedicine: A study on clinical Named Entity Recognition
  2. 2025Retrieval augmented generation based dynamic prompting for few-shot biomedical named entity recognition using large language models
  3. 2024LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction
  4. 2025Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings
  5. 2023How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain
  6. 2023Inspire the Large Language Model by External Knowledge on BioMedical Named Entity Recognition
  7. 2024Distilling Large Language Models for Efficient Clinical Information Extraction
  8. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing