# GPT-4 passes most of the 297 written Polish Board Certification Examinations

arXiv: https://arxiv.org/abs/2405.01589 · HTML: https://mlsanity.com/paper/2405.01589/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors](https://mlsanity.com/paper/2303.17807/) (2023)

## Built on by (most cited first, 10 of 10)

- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)
- [Polish-English medical knowledge transfer: A new benchmark and results](https://mlsanity.com/paper/2412.00559/) (2024)
- [LLMzSzL: a comprehensive LLM benchmark for Polish](https://mlsanity.com/paper/2501.02266/) (2025)
- [Bielik 11B v3: Multilingual Large Language Model for European Languages](https://mlsanity.com/paper/2601.11579/) (2025)
- [Bielik v3 Small: Technical Report](https://mlsanity.com/paper/2505.02550/) (2025)
- [Bielik 11B v2 Technical Report](https://mlsanity.com/paper/2505.02410/) (2025)
- [Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series](https://mlsanity.com/paper/2604.10799/) (2026)
- [Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?](https://mlsanity.com/paper/2606.12250/) (2026)
- [Optimizing Retrieval-Augmented Generation of Medical Content for Spaced Repetition Learning](https://mlsanity.com/paper/2503.01859/) (2025)
- [Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence](https://mlsanity.com/paper/2608.12928/) (2026)

## Abstract (arXiv)

Introduction: Recently, the effectiveness of Large Language Models (LLMs) has increased rapidly, allowing them to be used in a great number of applications. However, the risks posed by the generation of false information through LLMs significantly limit their applications in sensitive areas such as healthcare, highlighting the necessity for rigorous validations to determine their utility and reliability. To date, no study has extensively compared the performance of LLMs on Polish medical examinations across a broad spectrum of specialties on a very large dataset. Objectives: This study evaluated the performance of three Generative Pretrained Transformer (GPT) models on the Polish Board Certification Exam (Pa'nstwowy Egzamin Specjalizacyjny, PES) dataset, which consists of 297 tests. Methods: We developed a software program to download and process PES exams and tested the performance of GPT models using OpenAI Application Programming Interface. Results: Our findings reveal that GPT-3.5 did not pass any of the analyzed exams. In contrast, the GPT-4 models demonstrated the capability to pass the majority of the exams evaluated, with the most recent model, gpt-4-0125, successfully passing 222 (75%) of them. The performance of the GPT models varied significantly, displaying excellence in exams related to certain specialties while completely failing others. Conclusions: The significant progress and impressive performance of LLM models hold great promise for the increased application of AI in the field of medicine in Poland. For instance, this advancement could lead to the development of AI-based medical assistants for healthcare professionals, enhancing the efficiency and accuracy of medical services.