mlsanityData to 14 Oct 2026

A Family of Pretrained Transformer Language Models for Russian

arXiv 2309.10931 ↗First posted 19 Sep 2023cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 48 of its 88 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

48 of 88 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
36 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
  13. 2022UL2: Unifying Language Learning Paradigmscited by 295
  14. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  15. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  16. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  17. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  18. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 149
  19. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  20. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  21. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  22. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  23. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  24. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 75
  25. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  26. 2020A Survey on Contextual Embeddingscited by 48
  27. 2022Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russiancited by 29
  28. 2020How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Contextcited by 26
  29. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  30. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  31. 2020DaNetQA: a yes/no Question Answering Dataset for the Russian Languagecited by 8
  32. 2022RuNNE-2022 Shared Task: Recognizing Nested Named Entitiescited by 8
  33. 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
  34. 2021Detecting Inappropriate Messages on Sensitive Topics that Could Harm a Company's Reputationcited by 6
  35. 2017Human and Machine Judgements for Russian Semantic Relatednesscited by 4
  36. 2022RuArg-2022: Argument Mining Evaluationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 12; 2026: 7.

27 papers.

Sort:

By area:

  1. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  2. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  3. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  4. 2024Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russiancited by 12
  5. 2024Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategycited by 7
  6. 2024RuBia: A Russian Language Bias Detection Datasetcited by 4
  7. 2024Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Textscited by 3
  8. 2025REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilitiescited by 3
  9. 2025McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Modelscited by 3
  10. 2025Building Russian Benchmark for Evaluation of Information Retrieval Modelscited by 2
  11. 2025QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgmentscited by 2
  12. 2025LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Textcited by 2
15 more
  1. 2025T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playgroundcited by 2
  2. 2024Exploring Fine-tuned Generative Models for Keyphrase Selection: A Case Study for Russiancited by 1
  3. 2025Multimodal Evaluation of Russian-language Architecturescited by 1
  4. 2026From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mixcited by 1
  5. 2024Exploring the Effectiveness of Methods for Persona Extractioncited by 0
  6. 2025Methods for Recognizing Nested Termscited by 0
  7. 2025Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntaxcited by 0
  8. 2025Wikipedia-based Datasets in Russian Information Retrieval Benchmark RusBEIRcited by 0
  9. 2025FiMMIA: scaling semantic perturbation-based membership inference across modalitiescited by 0
  10. 2026MOSLD-Bench: Multilingual Open-Set Learning and Discovery Benchmark for Text Categorizationcited by 0
  11. 2026Learning Nested Named Entity Recognition from Flat Annotationscited by 0
  12. 2026AI Appeals Processor: A Deep Learning Approach to Automated Classification of Citizen Appeals in Government Servicescited by 0
  13. 2026MKJ at SemEval-2026 Task 9: A Comparative Study of Generalist, Specialist, and Ensemble Strategies for Multilingual Polarizationcited by 0
  14. 2026Zero-Shot to Full-Resource: Cross-lingual Transfer Strategies for Aspect-Based Sentiment Analysiscited by 0
  15. 2026Complexity-Guided Component-wise Initialization for Language Model Pretrainingcited by 0