mlsanityData to 7 Oct 2026

Evaluation of Transfer Learning for Polish with a Text-to-Text Model

arXiv 2205.08808 ↗First posted 18 May 2022cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, and question answering. In particular, since summarization and question answering lack benchmark datasets for the Polish language, we describe their construction and make them publicly available. Additionally, we present plT5 - a general-purpose text-to-text model for Polish that can be fine-tuned on various Natural Language Processing (NLP) tasks with a single training objective. Unsupervised denoising pre-training is performed efficiently by initializing the model weights with a multi-lingual T5 (mT5) counterpart. We evaluate the performance of plT5, mT5, Polish BART (plBART), and Polish GPT-2 (papuGaPT2). The plT5 scores top on all of these tasks except summarization, where plBART is best. In general (except for summarization), the larger the model, the better the results. The encoder-decoder architectures prove to be better than the decoder-only equivalent.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

28 of 45 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
16 more
  1. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  2. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  3. 2019CamemBERT: a Tasty French Language Modelcited by 368
  4. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  5. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  6. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  7. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  8. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  9. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  10. 2020German's Next Language Modelcited by 144
  11. 2019Multilingual is not enough: BERT for Finnishcited by 118
  12. 2020KLEJ: Comprehensive Benchmark for Polish Language Understandingcited by 51
  13. 2021HerBERT: Efficiently Pretrained Transformer-based Language Model for Polishcited by 51
  14. 2020A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasetscited by 40
  15. 2020PTT5: Pretraining and validating the T5 model on Brazilian Portuguese datacited by 35
  16. 2021IndT5: A Text-to-Text Transformer for 10 Indigenous Languagescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 6; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Languagecited by 15
  2. 2024ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Languagecited by 8
  3. 2022PolQA: Polish Question Answering Datasetcited by 7
  4. 2024A Survey of Large Language Models for European Languagescited by 6
  5. 2022Keyword Extraction from Short Texts with a Text-To-Text Transfer Transformercited by 4
  6. 2024Assessing generalization capability of text ranking models in Polishcited by 3
  7. 2024Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Constructioncited by 1
  8. 2025The PLLuM Instruction Corpuscited by 1
  9. 2026Long-Context Encoder Models for Polish Language Understandingcited by 1
  10. 2023Exploring the Use of Foundation Models for Named Entity Recognition and Lemmatization Tasks in Slavic Languagescited by 0
  11. 2024Two Approaches to Diachronic Normalization of Polish Textscited by 0
  12. 2024Passage Retrieval of Polish Texts Using OKAPI BM25 and an Ensemble of Cross Encoderscited by 0
1 more
  1. 2025AKCIT-FN at CheckThat! 2025: Switching Fine-Tuned SLMs and LLM Prompting for Multilingual Claim Normalizationcited by 0