# LLMzSzL: a comprehensive LLM benchmark for Polish

arXiv: https://arxiv.org/abs/2501.02266 · HTML: https://mlsanity.com/paper/2501.02266/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 4 Jan 2025.

## Builds on (most cited first, 13 of 13)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish](https://mlsanity.com/paper/2407.12402/) (2024)
- [Spanish and LLM Benchmarks: is MMLU Lost in Translation?](https://mlsanity.com/paper/2406.17789/) (2024)
- [GPT-4 passes most of the 297 written Polish Board Certification Examinations](https://mlsanity.com/paper/2405.01589/) (2024)
- [Polish-English medical knowledge transfer: A new benchmark and results](https://mlsanity.com/paper/2412.00559/) (2024)

## Built on by (most cited first, 7 of 7)

- [Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish](https://mlsanity.com/paper/2504.01667/) (2025)
- [PL-Guard: Benchmarking Language Model Safety for Polish](https://mlsanity.com/paper/2506.16322/) (2025)
- [La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America](https://mlsanity.com/paper/2507.00999/) (2025)
- [Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?](https://mlsanity.com/paper/2606.12250/) (2026)
- [Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation](https://mlsanity.com/paper/2608.07763/) (2026)
- [Lost in Historical Time? A Polish History Matura Benchmark for Large Language Models](https://mlsanity.com/paper/2608.12343/) (2026)
- [Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence](https://mlsanity.com/paper/2608.12928/) (2026)

## Abstract (arXiv)

This article introduces the first comprehensive benchmark for the Polish language at this scale: LLMzSzL (LLMs Behind the School Desk). It is based on a coherent collection of Polish national exams, including both academic and professional tests extracted from the archives of the Polish Central Examination Board. It covers 4 types of exams, coming from 154 domains. Altogether, it consists of almost 19k closed-ended questions. We investigate the performance of open-source multilingual, English, and Polish LLMs to verify LLMs' abilities to transfer knowledge between languages. Also, the correlation between LLMs and humans at model accuracy and exam pass rate levels is examined. We show that multilingual LLMs can obtain superior results over monolingual ones; however, monolingual models may be beneficial when model size matters. Our analysis highlights the potential of LLMs in assisting with exam validation, particularly in identifying anomalies or errors in examination tasks.