# Evaluation of Transfer Learning for Polish with a Text-to-Text Model

arXiv: https://arxiv.org/abs/2205.08808 · HTML: https://mlsanity.com/paper/2205.08808/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 18 May 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Mesh-TensorFlow: Deep Learning for Supercomputers](https://mlsanity.com/paper/1811.02084/) (2018)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish](https://mlsanity.com/paper/2105.01735/) (2021)
- [A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets](https://mlsanity.com/paper/2006.11880/) (2020)
- [PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data](https://mlsanity.com/paper/2008.09144/) (2020)
- [IndT5: A Text-to-Text Transformer for 10 Indigenous Languages](https://mlsanity.com/paper/2104.07483/) (2021)

## Built on by (most cited first, 13 of 13)

- [BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Language](https://mlsanity.com/paper/2305.19840/) (2023)
- [ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language](https://mlsanity.com/paper/2406.10806/) (2024)
- [PolQA: Polish Question Answering Dataset](https://mlsanity.com/paper/2212.08897/) (2022)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Keyword Extraction from Short Texts with a Text-To-Text Transfer Transformer](https://mlsanity.com/paper/2209.14008/) (2022)
- [Assessing generalization capability of text ranking models in Polish](https://mlsanity.com/paper/2402.14318/) (2024)
- [Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction](https://mlsanity.com/paper/2408.02337/) (2024)
- [The PLLuM Instruction Corpus](https://mlsanity.com/paper/2511.17161/) (2025)
- [Long-Context Encoder Models for Polish Language Understanding](https://mlsanity.com/paper/2603.12191/) (2026)
- [Exploring the Use of Foundation Models for Named Entity Recognition and Lemmatization Tasks in Slavic Languages](https://mlsanity.com/paper/2304.05336/) (2023)
- [Two Approaches to Diachronic Normalization of Polish Texts](https://mlsanity.com/paper/2402.01300/) (2024)
- [Passage Retrieval of Polish Texts Using OKAPI BM25 and an Ensemble of Cross Encoders](https://mlsanity.com/paper/2410.04620/) (2024)
- [AKCIT-FN at CheckThat! 2025: Switching Fine-Tuned SLMs and LLM Prompting for Multilingual Claim Normalization](https://mlsanity.com/paper/2509.11496/) (2025)

## Abstract (arXiv)

We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, and question answering. In particular, since summarization and question answering lack benchmark datasets for the Polish language, we describe their construction and make them publicly available. Additionally, we present plT5 - a general-purpose text-to-text model for Polish that can be fine-tuned on various Natural Language Processing (NLP) tasks with a single training objective. Unsupervised denoising pre-training is performed efficiently by initializing the model weights with a multi-lingual T5 (mT5) counterpart. We evaluate the performance of plT5, mT5, Polish BART (plBART), and Polish GPT-2 (papuGaPT2). The plT5 scores top on all of these tasks except summarization, where plBART is best. In general (except for summarization), the larger the model, the better the results. The encoder-decoder architectures prove to be better than the decoder-only equivalent.