# PhoGPT: Generative Pre-training for Vietnamese

arXiv: https://arxiv.org/abs/2311.02945 · HTML: https://mlsanity.com/paper/2311.02945/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [Efficient Finetuning Large Language Models For Vietnamese Chatbot](https://mlsanity.com/paper/2309.04646/) (2023)

## Built on by (most cited first, 8 of 8)

- [VinaLLaMA: LLaMA-based Vietnamese Foundation Model](https://mlsanity.com/paper/2312.11011/) (2023)
- [Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models](https://mlsanity.com/paper/2403.02715/) (2024)
- [GreenMind: A Next-Generation Vietnamese Large Language Model for Structured and Logical Reasoning](https://mlsanity.com/paper/2504.16832/) (2025)
- [ComOM at VLSP 2023: A Dual-Stage Framework with BERTology and Unified Multi-Task Instruction Tuning Model for Vietnamese Comparative Opinion Mining](https://mlsanity.com/paper/2312.09000/) (2023)
- [ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models](https://mlsanity.com/paper/2404.11086/) (2024)
- [VietBinoculars: A Zero-Shot Approach for Detecting Vietnamese LLM-Generated Text](https://mlsanity.com/paper/2509.26189/) (2025)
- [ConnectED: A Curriculum-Aligned AI System for Vietnamese Instructional Lesson Planning and Student Learning](https://mlsanity.com/paper/2607.28647/) (2026)
- [VietAIDetector: An Open-Source Zero-Shot Detector for Vietnamese AI-Generated Text](https://mlsanity.com/paper/2608.25478/) (2026)

## Abstract (arXiv)

We open-source a state-of-the-art 4B-parameter generative model series for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens, with an 8192 context length, employing a vocabulary of 20480 token types. The chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along with an additional 290K conversations. In addition, we also demonstrate its superior performance compared to previous open-source models. Our PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT