mlsanityData to 7 Oct 2026

Evaluating Large Language Models for Radiology Natural Language Processing

arXiv 2307.13693 ↗First posted 25 Jul 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The rise of large language models (LLMs) has marked a pivotal shift in the field of natural language processing (NLP). LLMs have revolutionized a multitude of domains, and they have made a significant impact in the medical field. Large language models are now more abundant than ever, and many of these models exhibit bilingual capabilities, proficient in both English and Chinese. However, a comprehensive evaluation of these models remains to be conducted. This lack of assessment is especially apparent within the context of radiology NLP. This study seeks to bridge this gap by critically evaluating thirty two LLMs in interpreting radiology reports, a crucial component of radiology NLP. Specifically, the ability to derive impressions from radiologic findings is assessed. The outcomes of this evaluation provide key insights into the performance, strengths, and weaknesses of these LLMs, informing their practical applications within the medical domain.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 57 of its 142 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

57 of 142 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2022Galactica: A Large Language Model for Sciencecited by 736
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  6. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  7. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  8. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  10. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  11. 2022Can large language models reason about medical questions?cited by 194
  12. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  13. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  14. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  15. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  16. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  17. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  18. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  19. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  20. 2023An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPTcited by 57
  21. 2023When Brain-inspired AI Meets AGIcited by 54
  22. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  23. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45
  24. 2023Radiology-GPT: A Large Language Model for Radiologycited by 44
  25. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Modelscited by 42
  26. 2023Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understandingcited by 40
  27. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39
  28. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  29. 2023AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiologycited by 34
  30. 2023Context Matters: A Strategy to Pre-train Language Model for Science Educationcited by 32
  31. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Taskcited by 32
  32. 2023ChatABL: Abductive Learning via Natural Language Interaction with ChatGPTcited by 26
  33. 2023Mask-guided BERT for Few Shot Text Classificationcited by 25
  34. 2023Segment Anything Model (SAM) for Radiation Oncologycited by 24
  35. 2023Putting ChatGPT's Medical Advice to the (Turing) Testcited by 23
  36. 2023Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applicationscited by 21
  37. 2022Coarse-to-fine Knowledge Graph Domain Adaptation based on Distantly-supervised Iterative Trainingcited by 20
  38. 2023An Evaluation on Large Language Model Outputs: Discourse and Memorizationcited by 20

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 7.

12 papers.

  1. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  2. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  3. 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
  4. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  5. 2023RadOnc-GPT: A Large Language Model for Radiation Oncologycited by 18
  6. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15
  7. 2024Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarioscited by 10
  8. 2024Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levelscited by 9
  9. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  10. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reportscited by 6
  11. 2024The current status of large language models in summarizing radiology report impressionscited by 4
  12. 2024Integrating ChatGPT into Secure Hospital Networks: A Case Study on Improving Radiology Report Analysiscited by 1