# RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model

arXiv: https://arxiv.org/abs/2105.11314 · HTML: https://mlsanity.com/paper/2105.11314/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [Neural Architectures for Nested NER through Linearization](https://mlsanity.com/paper/1908.06926/) (2019)
- [Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://mlsanity.com/paper/1508.02096/) (2015)
- [Czert -- Czech BERT-like Model for Language Representation](https://mlsanity.com/paper/2103.13031/) (2021)
- ['UFAL at MRP 2020: Permutation-invariant Semantic Parsing in PERIN](https://mlsanity.com/paper/2011.00758/) (2020)
- [LemmaTag: Jointly Tagging and Lemmatizing for Morphologically-Rich Languages with BRNNs](https://mlsanity.com/paper/1808.03703/) (2018)
- [Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NER](https://mlsanity.com/paper/1909.03544/) (2019)
- [HUJI-KU at MRP~2020: Two Transition-based Neural Parsers](https://mlsanity.com/paper/2010.05710/) (2020)

## Built on by (most cited first, 30 of 33)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022)
- [Speaking Multiple Languages Affects the Moral Bias of Language Models](https://mlsanity.com/paper/2211.07733/) (2022)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [Do Multilingual Language Models Capture Differing Moral Norms?](https://mlsanity.com/paper/2203.09904/) (2022)
- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [Comparison of Czech Transformers on Text Classification Tasks](https://mlsanity.com/paper/2107.10042/) (2021)
- [L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi](https://mlsanity.com/paper/2211.11187/) (2022)
- ['UFAL CorPipe at CRAC 2022: Effectivity of Multilingual Models for Coreference Resolution](https://mlsanity.com/paper/2209.07278/) (2022)
- [Quality and Efficiency of Manual Annotation: Pre-annotation Bias](https://mlsanity.com/paper/2306.09307/) (2023)
- [Siamese BERT-based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset](https://mlsanity.com/paper/2112.01810/) (2021)
- [Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages](https://mlsanity.com/paper/2112.10553/) (2021)
- [CsFEVER and CTKFacts: Acquiring Czech data for fact verification](https://mlsanity.com/paper/2201.11115/) (2022)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Czech Dataset for Cross-lingual Subjectivity Classification](https://mlsanity.com/paper/2204.13915/) (2022)
- [Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Language](https://mlsanity.com/paper/2312.10171/) (2023)
- [Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks](https://mlsanity.com/paper/2508.08125/) (2025)
- [Some Like It Small: Czech Semantic Embedding Models for Industry Applications](https://mlsanity.com/paper/2311.13921/) (2023)
- [Unpacking Robustness in Inflectional Languages: Adversarial Evaluation and Mechanistic Insights](https://mlsanity.com/paper/2505.07856/) (2025)
- [Unsupervised extraction, labelling and clustering of segments from clinical notes](https://mlsanity.com/paper/2211.11799/) (2022)
- [A Dataset and Strong Baselines for Classification of Czech News Texts](https://mlsanity.com/paper/2307.10666/) (2023)
- [How Gender Interacts with Political Values: A Case Study on Czech BERT Models](https://mlsanity.com/paper/2403.13514/) (2024)
- [Open-Source Web Service with Morphological Dictionary-Supplemented Deep Learning for Morphosyntactic Analysis of Czech](https://mlsanity.com/paper/2406.12422/) (2024)
- [GPT Czech Poet: Generation of Czech Poetic Strophes with Language Models](https://mlsanity.com/paper/2407.12790/) (2024)
- [Prompt-Based Approach for Czech Sentiment Analysis](https://mlsanity.com/paper/2508.08651/) (2025)
- [Different Time, Different Language: Revisiting the Bias Against Non-Native Speakers in GPT Detectors](https://mlsanity.com/paper/2602.05769/) (2026)
- [BERTuit: Understanding Spanish language in Twitter through a native transformer](https://mlsanity.com/paper/2204.03465/) (2022)
- [AlbNER: A Corpus for Named Entity Recognition in Albanian](https://mlsanity.com/paper/2309.08741/) (2023)
- [L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models](https://mlsanity.com/paper/2508.21569/) (2025)
- [Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation](https://mlsanity.com/paper/2506.20203/) (2025)

## Abstract (arXiv)

We present RobeCzech, a monolingual RoBERTa language representation model trained on Czech data. RoBERTa is a robustly optimized Transformer-based pretraining approach. We show that RobeCzech considerably outperforms equally-sized multilingual and Czech-trained contextualized language representation models, surpasses current state of the art in all five evaluated NLP tasks and reaches state-of-the-art results in four of them. The RobeCzech model is released publicly at https://hdl.handle.net/11234/1-3691 and https://huggingface.co/ufal/robeczech-base.