# Polish-English medical knowledge transfer: A new benchmark and results

arXiv: https://arxiv.org/abs/2412.00559 · HTML: https://mlsanity.com/paper/2412.00559/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 30 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification](https://mlsanity.com/paper/2406.08660/) (2024)
- [Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://mlsanity.com/paper/2303.18027/) (2023)
- [GPT-4 passes most of the 297 written Polish Board Certification Examinations](https://mlsanity.com/paper/2405.01589/) (2024)
- [Do Large Language Models have Shared Weaknesses in Medical Question Answering?](https://mlsanity.com/paper/2310.07225/) (2023)

## Built on by (most cited first, 8 of 8)

- [LLMzSzL: a comprehensive LLM benchmark for Polish](https://mlsanity.com/paper/2501.02266/) (2025)
- [Evaluating Polish linguistic and cultural competency in large language models](https://mlsanity.com/paper/2503.00995/) (2025)
- [Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?](https://mlsanity.com/paper/2606.12250/) (2026)
- [Optimizing Retrieval-Augmented Generation of Medical Content for Spaced Repetition Learning](https://mlsanity.com/paper/2503.01859/) (2025)
- [Patterns vs. Patients: Evaluating LLMs against Mental Health Professionals on Personality Disorder Diagnosis through First-Person Narratives](https://mlsanity.com/paper/2512.20298/) (2025)
- [EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation](https://mlsanity.com/paper/2604.14306/) (2026)
- [Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence](https://mlsanity.com/paper/2608.12928/) (2026)
- [Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics](https://mlsanity.com/paper/2609.09070/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated significant potential in handling specialized tasks, including medical problem-solving. However, most studies predominantly focus on English-language contexts. This study introduces a novel benchmark dataset based on Polish medical licensing and specialization exams (LEK, LDEK, PES) taken by medical doctor candidates and practicing doctors pursuing specialization. The dataset was web-scraped from publicly available resources provided by the Medical Examination Center and the Chief Medical Chamber. It comprises over 24,000 exam questions, including a subset of parallel Polish-English corpora, where the English portion was professionally translated by the examination center for foreign candidates. By creating a structured benchmark from these existing exam questions, we systematically evaluate state-of-the-art LLMs, including general-purpose, domain-specific, and Polish-specific models, and compare their performance against human medical students. Our analysis reveals that while models like GPT-4o achieve near-human performance, significant challenges persist in cross-lingual translation and domain-specific understanding. These findings underscore disparities in model performance across languages and medical specialties, highlighting the limitations and ethical considerations of deploying LLMs in clinical practice.