# Comparison of Czech Transformers on Text Classification Tasks

arXiv: https://arxiv.org/abs/2107.10042 · HTML: https://mlsanity.com/paper/2107.10042/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [DocBERT: BERT for Document Classification](https://mlsanity.com/paper/1904.08398/) (2019)
- [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021)
- [Czech Text Document Corpus v 2.0](https://mlsanity.com/paper/1710.02365/) (2017)
- [Deep Neural Networks for Czech Multi-label Document Classification](https://mlsanity.com/paper/1701.03849/) (2017)

## Built on by (most cited first, 10 of 10)

- [CsFEVER and CTKFacts: Acquiring Czech data for fact verification](https://mlsanity.com/paper/2201.11115/) (2022)
- [Exploring Capabilities of Monolingual Audio Transformers using Large Datasets in Automatic Speech Recognition of Czech](https://mlsanity.com/paper/2206.07627/) (2022)
- [Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks](https://mlsanity.com/paper/2508.08125/) (2025)
- [Some Like It Small: Czech Semantic Embedding Models for Industry Applications](https://mlsanity.com/paper/2311.13921/) (2023)
- [skLEP: A Slovak General Language Understanding Benchmark](https://mlsanity.com/paper/2506.21508/) (2025)
- [A Dataset and Strong Baselines for Classification of Czech News Texts](https://mlsanity.com/paper/2307.10666/) (2023)
- [How Gender Interacts with Political Values: A Case Study on Czech BERT Models](https://mlsanity.com/paper/2403.13514/) (2024)
- [CWRCzech: 100M Query-Document Czech Click Dataset and Its Application to Web Relevance Ranking](https://mlsanity.com/paper/2405.20994/) (2024)
- [Prompt-Based Approach for Czech Sentiment Analysis](https://mlsanity.com/paper/2508.08651/) (2025)
- [Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation](https://mlsanity.com/paper/2506.20203/) (2025)

## Abstract (arXiv)

In this paper, we present our progress in pre-training monolingual Transformers for Czech and contribute to the research community by releasing our models for public. The need for such models emerged from our effort to employ Transformers in our language-specific tasks, but we found the performance of the published multilingual models to be very limited. Since the multilingual models are usually pre-trained from 100+ languages, most of low-resourced languages (including Czech) are under-represented in these models. At the same time, there is a huge amount of monolingual training data available in web archives like Common Crawl. We have pre-trained and publicly released two monolingual Czech Transformers and compared them with relevant public models, trained (at least partially) for Czech. The paper presents the Transformers pre-training procedure as well as a comparison of pre-trained models on text classification task from various domains.