# Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training

arXiv: https://arxiv.org/abs/2403.15470 · HTML: https://mlsanity.com/paper/2403.15470/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 5 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 1 of 1)

- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)

## Built on by (most cited first, 4 of 4)

- [From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages](https://mlsanity.com/paper/2410.18836/) (2024)
- [Using Large Language Models for education managements in Vietnamese with low resources](https://mlsanity.com/paper/2501.15022/) (2025)
- [UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings](https://mlsanity.com/paper/2502.16961/) (2025)
- [VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP](https://mlsanity.com/paper/2608.03095/) (2026)

## Abstract (arXiv)

The advancement of Large Language Models (LLMs) has significantly transformed the field of natural language processing, although the focus on English-centric models has created a noticeable research gap for specific languages, including Vietnamese. To address this issue, this paper presents vi-mistral-x, an innovative Large Language Model designed expressly for the Vietnamese language. It utilizes a unique method of continual pre-training, based on the Mistral architecture, which incorporates grouped-query attention and sliding window attention techniques. This model, vi-Mistral-X, marks a significant step forward in improving the understanding and generation of the Vietnamese language. It introduces an additional phase of continual pre-training, specifically adapted for Vietnamese, enhancing the model's capability in understanding complex language nuances and generating accurate, context-aware Vietnamese text. Through comprehensive testing on various benchmarks, vi-mistral-x has shown to outperform existing Vietnamese LLMs in several key areas, including text classification, question answering, and text generation. Particularly, in the Vietnamese Multitask Language Understanding (VMLU) benchmark, vi-mistral-x sets a new standard, outperforming other available models significantly. This paper highlights the critical role of continual pre-training in advancing language-specific LLMs and opens new avenues for the development of multilingual models. We aim for vi-mistral-x to not just be an important asset for processing the Vietnamese language but also to encourage more advancements in creating large language models for languages that are less represented.