mlsanityData to 7 Oct 2026

GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

arXiv 2506.09440 ↗First posted 11 Jun 2025cs.CL cs.AIcited by 8

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open-source (https://huggingface.co/ai-sage), aiming to expand NLP research opportunities and support the development of industrial solutions for the Russian language.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 22 of its 39 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

22 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free14 Oct
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
7 more
  1. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  2. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  3. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  4. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  5. 2024Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Frameworkcited by 25
  6. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  7. 2020RuBQ: A Russian Dataset for Question Answering over Wikidatacited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2026Green-VLA: Staged Vision-Language-Action Model for Generalist Robotscited by 6
  2. 2025MERA Code: A Unified Framework for Evaluating Code Generation Across Taskscited by 4
  3. 2025HAMSA: Hijacking Aligned Compact Models via Stealthy Automationcited by 2
  4. 2025LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Textcited by 2
  5. 2025LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Datacited by 1
  6. 2025AINL-Eval 2025 Shared Task: Detection of AI-Generated Scientific Abstracts in Russiancited by 0
  7. 2025Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLMcited by 0
  8. 2026From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mixcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models
  2. 2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applications
  3. 2024A Survey of Large Language Models for European Languages
  4. 2023A Comprehensive Overview of Large Language Models
  5. 2024OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models
  6. 2024Data-Augmentation-Based Dialectal Adaptation for LLMs
  7. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems