# HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

arXiv: https://arxiv.org/abs/2105.01735 · HTML: https://mlsanity.com/paper/2105.01735/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018)
- [Pre-training Polish Transformer-based Language Models at Scale](https://mlsanity.com/paper/2006.04229/) (2020)

## Built on by (most cited first, 30 of 51)

- [PL-MTEB: Polish Massive Text Embedding Benchmark](https://mlsanity.com/paper/2405.10138/) (2024)
- [BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Language](https://mlsanity.com/paper/2305.19840/) (2023)
- [PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods](https://mlsanity.com/paper/2402.13350/) (2024)
- [Evaluation of Transfer Learning for Polish with a Text-to-Text Model](https://mlsanity.com/paper/2205.08808/) (2022)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases](https://mlsanity.com/paper/2207.12759/) (2022)
- [MAUPQA: Massive Automatically-created Polish Question Answering Dataset](https://mlsanity.com/paper/2305.05486/) (2023)
- ['UFAL CorPipe at CRAC 2022: Effectivity of Multilingual Models for Coreference Resolution](https://mlsanity.com/paper/2209.07278/) (2022)
- [PolQA: Polish Question Answering Dataset](https://mlsanity.com/paper/2212.08897/) (2022)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Development of Pre-Trained Transformer-based Models for the Nepali Language](https://mlsanity.com/paper/2411.15734/) (2024)
- [Silver Retriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://mlsanity.com/paper/2309.08469/) (2023)
- [End-to-end Multilingual Coreference Resolution with Mention Head Prediction](https://mlsanity.com/paper/2209.12516/) (2022)
- [Assessing generalization capability of text ranking models in Polish](https://mlsanity.com/paper/2402.14318/) (2024)
- [Exploring Multiple Strategies to Improve Multilingual Coreference Resolution in CorefUD](https://mlsanity.com/paper/2408.16893/) (2024)
- [Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian](https://mlsanity.com/paper/2501.02631/) (2025)
- [Unpacking Robustness in Inflectional Languages: Adversarial Evaluation and Mechanistic Insights](https://mlsanity.com/paper/2505.07856/) (2025)
- [COMBO: State-of-the-Art Morphosyntactic Analysis](https://mlsanity.com/paper/2109.05361/) (2021)
- [Going beyond research datasets: Novel intent discovery in the industry setting](https://mlsanity.com/paper/2305.05474/) (2023)
- [Towards Model-Based Data Acquisition for Subjective Multi-Task NLP Problems](https://mlsanity.com/paper/2312.08198/) (2023)
- [Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models](https://mlsanity.com/paper/2506.07645/) (2025)
- [PL-Guard: Benchmarking Language Model Safety for Polish](https://mlsanity.com/paper/2506.16322/) (2025)
- [COMBO: a new module for EUD parsing](https://mlsanity.com/paper/2107.03809/) (2021)
- [Detection of Criminal Texts for the Polish State Border Guard](https://mlsanity.com/paper/2108.10580/) (2021)
- [Improving Domain-Specific Retrieval by NLI Fine-Tuning](https://mlsanity.com/paper/2308.03103/) (2023)
- [Contrastive News and Social Media Linking using BERT for Articles and Tweets across Dual Platforms](https://mlsanity.com/paper/2312.07599/) (2023)
- [Evaluating Shortest Edit Script Methods for Contextual Lemmatization](https://mlsanity.com/paper/2403.16968/) (2024)
- [POLygraph: Polish Fake News Dataset](https://mlsanity.com/paper/2407.01393/) (2024)
- [Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction](https://mlsanity.com/paper/2408.02337/) (2024)
- [Multilingual hierarchical classification of job advertisements for job vacancy statistics](https://mlsanity.com/paper/2411.03779/) (2024)

## Abstract (arXiv)

BERT-based models are currently used for solving nearly all Natural Language Processing (NLP) tasks and most often achieve state-of-the-art results. Therefore, the NLP community conducts extensive research on understanding these models, but above all on designing effective and efficient training procedures. Several ablation studies investigating how to train BERT-like models have been carried out, but the vast majority of them concerned only the English language. A training procedure designed for English does not have to be universal and applicable to other especially typologically different languages. Therefore, this paper presents the first ablation study focused on Polish, which, unlike the isolating English language, is a fusional language. We design and thoroughly evaluate a pretraining procedure of transferring knowledge from multilingual to monolingual BERT-based models. In addition to multilingual model initialization, other factors that possibly influence pretraining are also explored, i.e. training objective, corpus size, BPE-Dropout, and pretraining length. Based on the proposed procedure, a Polish BERT-based language model -- HerBERT -- is trained. This model achieves state-of-the-art results on multiple downstream tasks.