mlsanityData to 7 Oct 2026

Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA

arXiv 2603.04033 ↗First posted 4 Mar 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Automatic evaluation of medical open-ended question answering (OEQA) remains challenging due to the need for expert annotations. We evaluate whether large language models (LLMs) can act as judges of semantic equivalence in French medical OEQA, comparing closed-access, general-purpose, and biomedical domain-adapted models. Our results show that LLM-based judgments are strongly influenced by the model that generated the answer, with agreement varying substantially across generators. Domain-adapted and large general-purpose models achieve the highest alignment with expert annotations. We further show that lightweight adaptation of a compact model using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) substantially improves performance and reduces generator sensitivity, even with limited data. Overall, our findings highlight the need for generator-aware evaluation and suggest that carefully adapted small models can support scalable evaluation in low-resource medical settings.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 4 Mar 2026.

Builds on

16 of 27 references.

References · 16

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  10. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 102
  11. 2024Towards Leveraging Large Language Models for Automated Medical Q&A Evaluationcited by 14
  12. 2025Can GRPO Boost Complex Multimodal Table Understanding?cited by 9
4 more
  1. 2025MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answeringcited by 8
  2. 2025MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluationcited by 5
  3. 2025Reasoning LLMs in the Medical Domain: A Literature Surveycited by 5
  4. 2024EasyJudge: an Easy-to-use Tool for Comprehensive Response Evaluation of LLMscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounterscited by 1
  2. 2026Trade-offs in Medical LLM Adaptation: An Empirical Study in French QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
  2. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  3. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
  4. 2024The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models
  5. 2024OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models
  6. 2026To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models
  7. 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
  8. 2025Structured Outputs Enable General-Purpose LLMs to be Medical Experts