# PhoBERT: Pre-trained language models for Vietnamese

arXiv: https://arxiv.org/abs/2003.00744 · HTML: https://mlsanity.com/paper/2003.00744/

## Key facts

- Cited by 149 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 30 of 149)

- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation](https://mlsanity.com/paper/2205.06457/) (2022)
- [MarIA: Spanish Language Models](https://mlsanity.com/paper/2107.07253/) (2021)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [Typhoon: Thai Large Language Models](https://mlsanity.com/paper/2312.13951/) (2023)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [A Family of Pretrained Transformer Language Models for Russian](https://mlsanity.com/paper/2309.10931/) (2023)
- [MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2212.13492/) (2022)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [Intent Detection and Slot Filling for Vietnamese](https://mlsanity.com/paper/2104.02021/) (2021)
- [COVID-19 Named Entity Recognition for Vietnamese](https://mlsanity.com/paper/2104.03879/) (2021)
- [Sabi'a: Portuguese Large Language Models](https://mlsanity.com/paper/2304.07880/) (2023)
- [Constructive and Toxic Speech Detection for Open-domain Social Media Comments in Vietnamese](https://mlsanity.com/paper/2103.10069/) (2021)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [BERTifying Sinhala -- A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classification](https://mlsanity.com/paper/2208.07864/) (2022)
- [XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech](https://mlsanity.com/paper/2305.19709/) (2023)
- [Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUT](https://mlsanity.com/paper/2404.09296/) (2024)
- [PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing](https://mlsanity.com/paper/2101.01476/) (2021)
- [A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models](https://mlsanity.com/paper/2107.03844/) (2021)
- [ViHOS: Hate Speech Spans Detection for Vietnamese](https://mlsanity.com/paper/2301.10186/) (2023)
- [ViDeBERTa: A powerful pre-trained language model for Vietnamese](https://mlsanity.com/paper/2301.10439/) (2023)
- [ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing](https://mlsanity.com/paper/2310.11166/) (2023)
- [Sentence Extraction-Based Machine Reading Comprehension for Vietnamese](https://mlsanity.com/paper/2105.09043/) (2021)
- [A Summary of the ALQAC 2021 Competition](https://mlsanity.com/paper/2204.10717/) (2022)

## Abstract (arXiv)

We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual language models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent best pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. We release PhoBERT to facilitate future research and downstream applications for Vietnamese NLP. Our PhoBERT models are available at https://github.com/VinAIResearch/PhoBERT