# Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

arXiv: https://arxiv.org/abs/2602.14564 · HTML: https://mlsanity.com/paper/2602.14564/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 16 Feb 2026.

## Builds on (most cited first, 15 of 15)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Automatic Evaluation of Healthcare LLMs Beyond Question-Answering](https://mlsanity.com/paper/2502.06666/) (2025)
- [BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text](https://mlsanity.com/paper/2504.19467/) (2025)
- [Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies](https://mlsanity.com/paper/2501.08441/) (2025)
- [MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMs](https://mlsanity.com/paper/2506.22808/) (2025)
- [Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions](https://mlsanity.com/paper/2508.08192/) (2025)
- [MedLM: Exploring Language Models for Medical Question Answering Systems](https://mlsanity.com/paper/2401.11389/) (2024)
- [Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology](https://mlsanity.com/paper/2508.13192/) (2025)
- [The Potential of LLMs in Medical Education: Generating Questions and Answers for Qualification Exams](https://mlsanity.com/paper/2410.23769/) (2024)
- [Collaboration among Multiple Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2505.16648/) (2025)
- [Faithful Summarization of Consumer Health Queries: A Cross-Lingual Framework with LLMs](https://mlsanity.com/paper/2511.10768/) (2025)

## Built on by (most cited first, 2 of 2)

- [BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali](https://mlsanity.com/paper/2605.31483/) (2026)
- [Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali](https://mlsanity.com/paper/2610.19003/) (2026)

## Abstract (arXiv)

Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.