mlsanityData to 14 Oct 2026

LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation

arXiv 2404.00998 ↗First posted 1 Apr 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

23 of 33 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2023Mistral 7Bcited by 3,164
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2024Mixtral of Expertscited by 1,823
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
11 more
  1. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 337
  2. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
  3. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  4. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 100
  5. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 74
  6. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 72
  7. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  8. 2024OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language modelscited by 20
  9. 2024Large Language Model Meets Graph Neural Network in Knowledge Distillationcited by 14
  10. 2023RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extractioncited by 11
  11. 2023Radiology-Aware Model-Based Evaluation Metric for Report Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 3; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024MAIRA-2: Grounded Radiology Report Generationcited by 145
  2. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  3. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
  4. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 25
  5. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  6. 2026CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluationcited by 9
  7. 2026Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMscited by 3
  8. 2024Assessing the Impact of Conspiracy Theories Using Large Language Modelscited by 2
  9. 2024Improving Expert Radiology Report Summarization by Prompting Large Language Models with a Layperson Summarycited by 1
  10. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Taskscited by 1
  11. 2025Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rayscited by 1
  12. 2026ECG-LLM: Foundation Model for ECG-Based Cardiac Reasoningcited by 1
3 more
  1. 2026RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matchingcited by 1
  2. 2025Performance Assessment Strategies for Language Model Applications in Healthcarecited by 0
  3. 2025Semantic Similarity in Radiology Reports via LLMs and NERcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MRScore: Evaluating Radiology Report Generation with LLM-based Reward System
  2. 2024Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports
  3. 2026ReportQA: QA-Based Radiology Report Evaluation
  4. 2024Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System
  5. 2024R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generation
  6. 2023Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation
  7. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
  8. 2025RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection