Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 10 Jun 2026.
Builds on
17 of 34 references.
References · 17
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
- 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
- 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
- 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
- 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
- 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
- 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
- 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
- 2024AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Datasetcited by 26
5 more
- 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 25
- 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
- 2024GPT-4 passes most of the 297 written Polish Board Certification Examinationscited by 10
- 2024Polish-English medical knowledge transfer: A new benchmark and resultscited by 8
- 2025LLMzSzL: a comprehensive LLM benchmark for Polishcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark
- 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset
- 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations
- 2025It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education
- 2024Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
- 2025Structured Outputs Enable General-Purpose LLMs to be Medical Experts
- 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
- 2026CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine