# VinaLLaMA: LLaMA-based Vietnamese Foundation Model

arXiv: https://arxiv.org/abs/2312.11011 · HTML: https://mlsanity.com/paper/2312.11011/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [PhoGPT: Generative Pre-training for Vietnamese](https://mlsanity.com/paper/2311.02945/) (2023)

## Built on by (most cited first, 10 of 10)

- [Sailor: Open Language Models for South-East Asia](https://mlsanity.com/paper/2404.03608/) (2024)
- [Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures](https://mlsanity.com/paper/2510.24081/) (2025)
- [BgGPT 1.0: Extending English-centric LLMs to other languages](https://mlsanity.com/paper/2412.10893/) (2024)
- [From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation](https://mlsanity.com/paper/2404.09138/) (2024)
- [Soro: A Lightweight Foundation Model and Chatbot for Tajik](https://mlsanity.com/paper/2605.27379/) (2026)
- [ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models](https://mlsanity.com/paper/2404.11086/) (2024)
- [A Benchmark Dataset and Evaluation Framework for Vietnamese Large Language Models in Customer Support](https://mlsanity.com/paper/2507.22542/) (2025)
- [ViMultiChoice: Toward a Method That Gives Explanation for Multiple-Choice Reading Comprehension in Vietnamese](https://mlsanity.com/paper/2602.09961/) (2026)
- [VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP](https://mlsanity.com/paper/2608.03095/) (2026)
- [CantoneseLLM v2: Reasoning in a Low-Resource Language](https://mlsanity.com/paper/2609.06970/) (2026)

## Abstract (arXiv)

In this technical report, we present VinaLLaMA, an open-weight, state-of-the-art (SOTA) Large Language Model for the Vietnamese language, built upon LLaMA-2 with an additional 800 billion trained tokens. VinaLLaMA not only demonstrates fluency in Vietnamese but also exhibits a profound understanding of Vietnamese culture, making it a truly indigenous model. VinaLLaMA-7B-chat, trained on 1 million high-quality synthetic samples, achieves SOTA results on key benchmarks, including VLSP, VMLU, and Vicuna Benchmark Vietnamese, marking a significant advancement in the Vietnamese AI landscape and offering a versatile resource for various applications.