# Pre-training Polish Transformer-based Language Models at Scale

arXiv: https://arxiv.org/abs/2006.04229 · HTML: https://mlsanity.com/paper/2006.04229/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022)
- [HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish](https://mlsanity.com/paper/2105.01735/) (2021)
- [PL-MTEB: Polish Massive Text Embedding Benchmark](https://mlsanity.com/paper/2405.10138/) (2024)
- [BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Language](https://mlsanity.com/paper/2305.19840/) (2023)
- [PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods](https://mlsanity.com/paper/2402.13350/) (2024)
- [SlovakBERT: Slovak Masked Language Model](https://mlsanity.com/paper/2109.15254/) (2021)
- [Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases](https://mlsanity.com/paper/2207.12759/) (2022)
- [Measuring Gender Bias in West Slavic Language Models](https://mlsanity.com/paper/2304.05783/) (2023)
- [BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service](https://mlsanity.com/paper/2308.10592/) (2023)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Assessing generalization capability of text ranking models in Polish](https://mlsanity.com/paper/2402.14318/) (2024)
- [LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma](https://mlsanity.com/paper/2403.00418/) (2024)
- [Detection of Criminal Texts for the Polish State Border Guard](https://mlsanity.com/paper/2108.10580/) (2021)
- [Long-Context Encoder Models for Polish Language Understanding](https://mlsanity.com/paper/2603.12191/) (2026)
- [Applying SoftTriple Loss for Supervised Language Model Fine Tuning](https://mlsanity.com/paper/2112.08462/) (2021)
- [AlcLaM: Arabic Dialectal Language Model](https://mlsanity.com/paper/2407.13097/) (2024)
- [Passage Retrieval of Polish Texts Using OKAPI BM25 and an Ensemble of Cross Encoders](https://mlsanity.com/paper/2410.04620/) (2024)
- [Punctuation Prediction for Polish Texts using Transformers](https://mlsanity.com/paper/2410.04621/) (2024)
- [Unveiling Dual Quality in Product Reviews: An NLP-Based Approach](https://mlsanity.com/paper/2505.19254/) (2025)
- [Polish ModernBERT: The Long and Short of Polish Language Understanding](https://mlsanity.com/paper/2609.01379/) (2026)

## Abstract (arXiv)

Transformer-based language models are now widely used in Natural Language Processing (NLP). This statement is especially true for English language, in which many pre-trained models utilizing transformer-based architecture have been published in recent years. This has driven forward the state of the art for a variety of standard NLP tasks such as classification, regression, and sequence labeling, as well as text-to-text tasks, such as machine translation, question answering, or summarization. The situation have been different for low-resource languages, such as Polish, however. Although some transformer-based language models for Polish are available, none of them have come close to the scale, in terms of corpus size and the number of parameters, of the largest English-language models. In this study, we present two language models for Polish based on the popular BERT architecture. The larger model was trained on a dataset consisting of over 1 billion polish sentences, or 135GB of raw text. We describe our methodology for collecting the data, preparing the corpus, and pre-training the model. We then evaluate our models on thirteen Polish linguistic tasks, and demonstrate improvements over previous approaches in eleven of them.