# VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding

arXiv: https://arxiv.org/abs/2403.15882 · HTML: https://mlsanity.com/paper/2403.15882/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 50 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [A Vietnamese Dataset for Evaluating Machine Reading Comprehension](https://mlsanity.com/paper/2009.14725/) (2020)
- [Emotion Recognition for Vietnamese Social Media Text](https://mlsanity.com/paper/1911.09339/) (2019)
- [New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles](https://mlsanity.com/paper/2006.11138/) (2020)
- [On the Multilingual Capabilities of Very Large-Scale English Language Models](https://mlsanity.com/paper/2108.13349/) (2021)
- [WikiBERT models: deep transfer learning for many languages](https://mlsanity.com/paper/2006.01538/) (2020)
- [Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks](https://mlsanity.com/paper/2108.08375/) (2021)

## Built on by (most cited first, 8 of 8)

- [R2GQA: Retriever-Reader-Generator Question Answering System to Support Students Understanding Legal Regulations in Higher Education](https://mlsanity.com/paper/2409.02840/) (2024)
- [Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese](https://mlsanity.com/paper/2411.13407/) (2024)
- [LiGT: Layout-infused Generative Transformer for Visual Question Answering on Vietnamese Receipts](https://mlsanity.com/paper/2502.19202/) (2025)
- [Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?](https://mlsanity.com/paper/2507.20419/) (2025)
- [ViToSA: Audio-Based Toxic Spans Detection on Vietnamese Speech Utterances](https://mlsanity.com/paper/2506.00636/) (2025)
- [ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks](https://mlsanity.com/paper/2603.21084/) (2026)
- [ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts](https://mlsanity.com/paper/2608.12776/) (2026)
- [Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining](https://mlsanity.com/paper/2609.21362/) (2026)

## Abstract (arXiv)

The success of Natural Language Understanding (NLU) benchmarks in various languages, such as GLUE for English, CLUE for Chinese, KLUE for Korean, and IndoNLU for Indonesian, has facilitated the evaluation of new NLU models across a wide range of tasks. To establish a standardized set of benchmarks for Vietnamese NLU, we introduce the first Vietnamese Language Understanding Evaluation (VLUE) benchmark. The VLUE benchmark encompasses five datasets covering different NLU tasks, including text classification, span extraction, and natural language understanding. To provide an insightful overview of the current state of Vietnamese NLU, we then evaluate seven state-of-the-art pre-trained models, including both multilingual and Vietnamese monolingual models, on our proposed VLUE benchmark. Furthermore, we present CafeBERT, a new state-of-the-art pre-trained model that achieves superior results across all tasks in the VLUE benchmark. Our model combines the proficiency of a multilingual pre-trained model with Vietnamese linguistic knowledge. CafeBERT is developed based on the XLM-RoBERTa model, with an additional pretraining step utilizing a significant amount of Vietnamese textual data to enhance its adaptation to the Vietnamese language. For the purpose of future research, CafeBERT is made publicly available for research purposes.