# Towards Leveraging Large Language Models for Automated Medical Q&A Evaluation

arXiv: https://arxiv.org/abs/2409.01941 · HTML: https://mlsanity.com/paper/2409.01941/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 31 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 1 of 1)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)

## Built on by (most cited first, 14 of 14)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning](https://mlsanity.com/paper/2506.09513/) (2025)
- [MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback](https://mlsanity.com/paper/2410.13191/) (2024)
- [Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards](https://mlsanity.com/paper/2601.08778/) (2026)
- [Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness](https://mlsanity.com/paper/2604.16383/) (2026)
- [Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA](https://mlsanity.com/paper/2603.04033/) (2026)
- [Developing and evaluating a chatbot to support maternal health care](https://mlsanity.com/paper/2603.13168/) (2026)
- [Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoring](https://mlsanity.com/paper/2604.18835/) (2026)
- [ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases](https://mlsanity.com/paper/2506.00095/) (2025)
- [The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses](https://mlsanity.com/paper/2511.00924/) (2025)
- [How Uncertain Is the Grade? A Benchmark of Uncertainty Metrics for LLM-Based Automatic Assessment](https://mlsanity.com/paper/2602.16039/) (2026)
- [COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning](https://mlsanity.com/paper/2608.30109/) (2026)
- [LaQual: An Automated Framework for LLM App Quality Evaluation](https://mlsanity.com/paper/2508.18636/) (2025)
- [How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting](https://mlsanity.com/paper/2601.11344/) (2026)

## Abstract (arXiv)

This paper explores the potential of using Large Language Models (LLMs) to automate the evaluation of responses in medical Question and Answer (Q&A) systems, a crucial form of Natural Language Processing. Traditionally, human evaluation has been indispensable for assessing the quality of these responses. However, manual evaluation by medical professionals is time-consuming and costly. Our study examines whether LLMs can reliably replicate human evaluations by using questions derived from patient data, thereby saving valuable time for medical experts. While the findings suggest promising results, further research is needed to address more specific or complex questions that were beyond the scope of this initial investigation.