# Czert -- Czech BERT-like Model for Language Representation arXiv: https://arxiv.org/abs/2103.13031 ยท HTML: https://mlsanity.com/paper/2103.13031/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 49 references that are in the map. - First posted on arXiv 24 Mar 2021. ## Builds on (most cited first, 25 of 25) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019) - [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019) - [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018) - [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019) - [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019) - [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019) - [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019) - [Learned in Translation: Contextualized Word Vectors](https://mlsanity.com/paper/1708.00107/) (2017) - [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020) - [The birth of Romanian BERT](https://mlsanity.com/paper/2009.08712/) (2020) - [Deep Neural Networks for Czech Multi-label Document Classification](https://mlsanity.com/paper/1701.03849/) (2017) - [Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NER](https://mlsanity.com/paper/1909.03544/) (2019) ## Built on by (most cited first, 22 of 22) - [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021) - [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023) - [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021) - [SlovakBERT: Slovak Masked Language Model](https://mlsanity.com/paper/2109.15254/) (2021) - [Measuring Gender Bias in West Slavic Language Models](https://mlsanity.com/paper/2304.05783/) (2023) - [Are the Multilingual Models Better? Improving Czech Sentiment with Transformers](https://mlsanity.com/paper/2108.10640/) (2021) - [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024) - [Czech Dataset for Cross-lingual Subjectivity Classification](https://mlsanity.com/paper/2204.13915/) (2022) - [Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Model](https://mlsanity.com/paper/2307.14785/) (2023) - [End-to-end Multilingual Coreference Resolution with Mention Head Prediction](https://mlsanity.com/paper/2209.12516/) (2022) - [Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Language](https://mlsanity.com/paper/2312.10171/) (2023) - [Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks](https://mlsanity.com/paper/2508.08125/) (2025) - [Some Like It Small: Czech Semantic Embedding Models for Industry Applications](https://mlsanity.com/paper/2311.13921/) (2023) - [Exploring Multiple Strategies to Improve Multilingual Coreference Resolution in CorefUD](https://mlsanity.com/paper/2408.16893/) (2024) - [How Gender Interacts with Political Values: A Case Study on Czech BERT Models](https://mlsanity.com/paper/2403.13514/) (2024) - [Prompt-Based Approach for Czech Sentiment Analysis](https://mlsanity.com/paper/2508.08651/) (2025) - [Czech News Dataset for Semantic Textual Similarity](https://mlsanity.com/paper/2108.08708/) (2021) - [Fine-grained Czech News Article Dataset: An Interdisciplinary Approach to Trustworthiness Analysis](https://mlsanity.com/paper/2212.08550/) (2022) - [Exploring the Use of Foundation Models for Named Entity Recognition and Lemmatization Tasks in Slavic Languages](https://mlsanity.com/paper/2304.05336/) (2023) - [TextBite: A Historical Czech Document Dataset for Logical Page Segmentation](https://mlsanity.com/paper/2503.16664/) (2025) - [Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation](https://mlsanity.com/paper/2506.20203/) (2025) - [CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents](https://mlsanity.com/paper/2603.03884/) (2026) ## Abstract (arXiv) This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.