# GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

arXiv: https://arxiv.org/abs/2506.09440 · HTML: https://mlsanity.com/paper/2506.09440/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 22 of 22)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free](https://mlsanity.com/paper/2410.10814/) (2024)
- [Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework](https://mlsanity.com/paper/2410.21061/) (2024)
- [MERA: A Comprehensive LLM Evaluation in Russian](https://mlsanity.com/paper/2401.04531/) (2024)
- [RuBQ: A Russian Dataset for Question Answering over Wikidata](https://mlsanity.com/paper/2005.10659/) (2020)

## Built on by (most cited first, 8 of 8)

- [Green-VLA: Staged Vision-Language-Action Model for Generalist Robots](https://mlsanity.com/paper/2602.00919/) (2026)
- [MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks](https://mlsanity.com/paper/2507.12284/) (2025)
- [HAMSA: Hijacking Aligned Compact Models via Stealthy Automation](https://mlsanity.com/paper/2508.16484/) (2025)
- [LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Text](https://mlsanity.com/paper/2509.21269/) (2025)
- [LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Data](https://mlsanity.com/paper/2507.13413/) (2025)
- [From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix](https://mlsanity.com/paper/2609.01572/) (2026)
- [AINL-Eval 2025 Shared Task: Detection of AI-Generated Scientific Abstracts in Russian](https://mlsanity.com/paper/2508.09622/) (2025)
- [Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM](https://mlsanity.com/paper/2512.21580/) (2025)

## Abstract (arXiv)

Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open-source (https://huggingface.co/ai-sage), aiming to expand NLP research opportunities and support the development of industrial solutions for the Russian language.