mlsanityData to 7 Oct 2026

Czert -- Czech BERT-like Model for Language Representation

arXiv 2103.13031 ↗First posted 24 Mar 2021cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 25 of its 49 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

25 of 49 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,578
13 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2019CamemBERT: a Tasty French Language Modelcited by 368
  4. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  5. 2019Simple BERT Models for Relation Extraction and Semantic Role Labelingcited by 154
  6. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  7. 2019Multilingual is not enough: BERT for Finnishcited by 118
  8. 2019BERTje: A Dutch BERT Modelcited by 109
  9. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  10. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  11. 2020The birth of Romanian BERTcited by 44
  12. 2017Deep Neural Networks for Czech Multi-label Document Classificationcited by 3
  13. 2019Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NERcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 3; 2023: 6; 2024: 3; 2025: 4; 2026: 1.

22 papers.

Sort:

  1. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  2. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  3. 2021RobeCzech: Czech RoBERTa, a monolingual contextualized language representation modelcited by 33
  4. 2021SlovakBERT: Slovak Masked Language Modelcited by 11
  5. 2023Measuring Gender Bias in West Slavic Language Modelscited by 10
  6. 2021Are the Multilingual Models Better? Improving Czech Sentiment with Transformerscited by 7
  7. 2024A Survey of Large Language Models for European Languagescited by 6
  8. 2022Czech Dataset for Cross-lingual Subjectivity Classificationcited by 5
  9. 2023Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Modelcited by 5
  10. 2022End-to-end Multilingual Coreference Resolution with Mention Head Predictioncited by 4
  11. 2023Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Languagecited by 4
  12. 2025Czech Dataset for Complex Aspect-Based Sentiment Analysis Taskscited by 4
10 more
  1. 2023Some Like It Small: Czech Semantic Embedding Models for Industry Applicationscited by 3
  2. 2024Exploring Multiple Strategies to Improve Multilingual Coreference Resolution in CorefUDcited by 3
  3. 2024How Gender Interacts with Political Values: A Case Study on Czech BERT Modelscited by 2
  4. 2025Prompt-Based Approach for Czech Sentiment Analysiscited by 2
  5. 2021Czech News Dataset for Semantic Textual Similaritycited by 1
  6. 2022Fine-grained Czech News Article Dataset: An Interdisciplinary Approach to Trustworthiness Analysiscited by 0
  7. 2023Exploring the Use of Foundation Models for Named Entity Recognition and Lemmatization Tasks in Slavic Languagescited by 0
  8. 2025TextBite: A Historical Czech Document Dataset for Logical Page Segmentationcited by 0
  9. 2025Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluationcited by 0
  10. 2026CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documentscited by 0