mlsanityData to 14 Oct 2026

SlovakBERT: Slovak Masked Language Model

arXiv 2109.15254 ↗First posted 30 Sep 2021cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce a new Slovak masked language model called SlovakBERT. This is to our best knowledge the first paper discussing Slovak transformers-based language models. We evaluate our model on several NLP tasks and achieve state-of-the-art results. This evaluation is likewise the first attempt to establish a benchmark for Slovak language models. We publish the masked language model, as well as the fine-tuned models for part-of-speech tagging, sentiment analysis and semantic textual similarity.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,847
25 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  8. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  11. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
  13. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  14. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  15. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  16. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  17. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  18. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  19. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  20. 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 38
  21. 2020Which *BERT? A Survey Organizing Contextualized Encoderscited by 26
  22. 2016Multilingual Twitter Sentiment Classification: The Role of Human Annotatorscited by 24
  23. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  24. 2020Pre-training Polish Transformer-based Language Models at Scalecited by 21
  25. 2020WikiBERT models: deep transfer learning for many languagescited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 2; 2025: 2; 2026: 1.

11 papers.

  1. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  2. 2023Measuring Gender Bias in West Slavic Language Modelscited by 10
  3. 2023In-Depth Look at Word Filling Societal Bias Measurescited by 8
  4. 2024A Survey of Large Language Models for European Languagescited by 6
  5. 2023Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Languagecited by 4
  6. 2025skLEP: A Slovak General Language Understanding Benchmarkcited by 3
  7. 2023Relationship between auditory and semantic entrainment using Deep Neural Networks (DNN)cited by 1
  8. 2023WikiGoldSK: Annotated Dataset, Baselines and Few-Shot Learning Experiments for Slovak Named Entity Recognitioncited by 0
  9. 2024Named Entity Recognition for Address Extraction in Speech-to-Text Transcriptions Using Synthetic Datacited by 0
  10. 2025Enhancing BERT Fine-Tuning for Sentiment Analysis in Lower-Resourced Languagescited by 0
  11. 2026Approaches to Semantic Textual Similarity in Slovak Language: From Algorithms to Transformerscited by 0