# Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

arXiv: https://arxiv.org/abs/2405.13929 · HTML: https://mlsanity.com/paper/2405.13929/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2010.15925/) (2020)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [PALO: A Polyglot Large Multimodal Model for 5B People](https://mlsanity.com/paper/2402.14818/) (2024)
- [A Family of Pretrained Transformer Language Models for Russian](https://mlsanity.com/paper/2309.10931/) (2023)
- [MIHash: Online Hashing with Mutual Information](https://mlsanity.com/paper/1703.08919/) (2017)
- [Impact of Tokenization on LLaMa Russian Adaptation](https://mlsanity.com/paper/2312.02598/) (2023)

## Built on by (most cited first, 12 of 12)

- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [NeuroStrike: Neuron-Level Attacks on Aligned LLMs](https://mlsanity.com/paper/2509.11864/) (2025)
- [PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation](https://mlsanity.com/paper/2409.06820/) (2024)
- [Facilitating large language model Russian adaptation with Learned Embedding Propagation](https://mlsanity.com/paper/2412.21140/) (2024)
- [Reliable, Reproducible, and Really Fast Leaderboards with Evalica](https://mlsanity.com/paper/2412.11314/) (2024)
- [Kuwain 1.5B: An Arabic SLM via Language Injection](https://mlsanity.com/paper/2504.15120/) (2025)
- [T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playground](https://mlsanity.com/paper/2512.10430/) (2025)
- [Multimodal Stock Price Prediction: A Case Study of the Russian Securities Market](https://mlsanity.com/paper/2503.08696/) (2025)
- [CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility](https://mlsanity.com/paper/2603.12872/) (2026)
- [From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix](https://mlsanity.com/paper/2609.01572/) (2026)
- [Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases](https://mlsanity.com/paper/2410.18040/) (2024)
- [Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX](https://mlsanity.com/paper/2505.24616/) (2025)

## Abstract (arXiv)

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. ``Vikhr'' refers to the name of the Mistral LLM series and means a ``strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.