# ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation

arXiv: https://arxiv.org/abs/2205.06457 · HTML: https://mlsanity.com/paper/2205.06457/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2010.03093/) (2020)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020)
- [SciFive: a text-to-text transformer model for biomedical literature](https://mlsanity.com/paper/2106.03598/) (2021)
- [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [Improving Sequence Tagging for Vietnamese Text Using Transformer-based Neural Models](https://mlsanity.com/paper/2006.15994/) (2020)
- [IT5: Text-to-text Pretraining for Italian Language Understanding and Generation](https://mlsanity.com/paper/2203.03759/) (2022)
- [COVID-19 Named Entity Recognition for Vietnamese](https://mlsanity.com/paper/2104.03879/) (2021)
- [Monolingual versus Multilingual BERTology for Vietnamese Extractive Multi-Document Summarization](https://mlsanity.com/paper/2108.13741/) (2021)
- [VieSum: How Robust Are Transformer-based Models on Vietnamese Summarization?](https://mlsanity.com/paper/2110.04257/) (2021)

## Built on by (most cited first, 30 of 37)

- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [ViDeBERTa: A powerful pre-trained language model for Vietnamese](https://mlsanity.com/paper/2301.10439/) (2023)
- [ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing](https://mlsanity.com/paper/2310.11166/) (2023)
- [Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi](https://mlsanity.com/paper/2504.06011/) (2025)
- [ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Images](https://mlsanity.com/paper/2404.18397/) (2024)
- [ViTextVQA: A Large-Scale Visual Question Answering Dataset and a Novel Multimodal Feature Fusion Method for Vietnamese Text Comprehension in Images](https://mlsanity.com/paper/2404.10652/) (2024)
- [Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models](https://mlsanity.com/paper/2403.02715/) (2024)
- [MTet: Multi-domain Translation for English and Vietnamese](https://mlsanity.com/paper/2210.05610/) (2022)
- [Enriching Biomedical Knowledge for Low-resource Language Through Large-Scale Translation](https://mlsanity.com/paper/2210.05598/) (2022)
- [ViWikiFC: Fact-Checking for Vietnamese Wikipedia-Based Textual Knowledge Source](https://mlsanity.com/paper/2405.07615/) (2024)
- [VLQA: The First Comprehensive, Large, and High-Quality Vietnamese Dataset for Legal Question Answering](https://mlsanity.com/paper/2507.19995/) (2025)
- [ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model](https://mlsanity.com/paper/2405.14141/) (2024)
- [FreeTxt-Vi: A Benchmarked Vietnamese-English Toolkit for Segmentation, Sentiment, and Summarisation](https://mlsanity.com/paper/2603.05690/) (2026)
- [Real-time Speech Summarization for Medical Conversations](https://mlsanity.com/paper/2406.15888/) (2024)
- [R2GQA: Retriever-Reader-Generator Question Answering System to Support Students Understanding Legal Regulations in Higher Education](https://mlsanity.com/paper/2409.02840/) (2024)
- [Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer](https://mlsanity.com/paper/2505.10945/) (2025)
- [Abstractive Text Summarization Using the BRIO Training Paradigm](https://mlsanity.com/paper/2305.13696/) (2023)
- [ComOM at VLSP 2023: A Dual-Stage Framework with BERTology and Unified Multi-Task Instruction Tuning Model for Vietnamese Comparative Opinion Mining](https://mlsanity.com/paper/2312.09000/) (2023)
- [VlogQA: Task, Dataset, and Baseline Models for Vietnamese Spoken-Based Machine Reading Comprehension](https://mlsanity.com/paper/2402.02655/) (2024)
- [Medical Spoken Named Entity Recognition](https://mlsanity.com/paper/2406.13337/) (2024)
- [A Large-Scale Benchmark for Vietnamese Sentence Paraphrases](https://mlsanity.com/paper/2502.07188/) (2025)
- [LiGT: Layout-infused Generative Transformer for Visual Question Answering on Vietnamese Receipts](https://mlsanity.com/paper/2502.19202/) (2025)
- [Advancements in Natural Language Processing for Automatic Text Summarization](https://mlsanity.com/paper/2502.19773/) (2025)
- [Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark](https://mlsanity.com/paper/2512.22218/) (2025)
- [LBMT team at VLSP2022-Abmusu: Hybrid method with text correlation and generative models for Vietnamese multi-document summarization](https://mlsanity.com/paper/2304.05205/) (2023)
- [Overview of the VLSP 2022 -- Abmusu Shared Task: A Data Challenge for Vietnamese Abstractive Multi-document Summarization](https://mlsanity.com/paper/2311.15525/) (2023)
- [Vietnamese AI Generated Text Detection](https://mlsanity.com/paper/2405.03206/) (2024)
- [Sentiment Reasoning for Healthcare](https://mlsanity.com/paper/2407.21054/) (2024)
- [ViConsFormer: Constituting Meaningful Phrases of Scene Texts using Transformer-based Method in Vietnamese Text-based Visual Question Answering](https://mlsanity.com/paper/2410.14132/) (2024)
- [Comparative Opinion Mining in Product Reviews: Multi-perspective Prompt-based Learning](https://mlsanity.com/paper/2412.08508/) (2024)

## Abstract (arXiv)

We present ViT5, a pretrained Transformer-based encoder-decoder model for the Vietnamese language. With T5-style self-supervised pretraining, ViT5 is trained on a large corpus of high-quality and diverse Vietnamese texts. We benchmark ViT5 on two downstream text generation tasks, Abstractive Text Summarization and Named Entity Recognition. Although Abstractive Text Summarization has been widely studied for the English language thanks to its rich and large source of data, there has been minimal research into the same task in Vietnamese, a much lower resource language. In this work, we perform exhaustive experiments on both Vietnamese Abstractive Summarization and Named Entity Recognition, validating the performance of ViT5 against many other pretrained Transformer-based encoder-decoder models. Our experiments show that ViT5 significantly outperforms existing models and achieves state-of-the-art results on Vietnamese Text Summarization. On the task of Named Entity Recognition, ViT5 is competitive against previous best results from pretrained encoder-based Transformer models. Further analysis shows the importance of context length during the self-supervised pretraining on downstream performance across different settings.