# ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing arXiv: https://arxiv.org/abs/2310.11166 ยท HTML: https://mlsanity.com/paper/2310.11166/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 41 references that are in the map. - First posted on arXiv 17 Oct 2023. ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019) - [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020) - [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020) - [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020) - [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022) - [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021) - [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter](https://mlsanity.com/paper/2209.07562/) (2022) - [ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation](https://mlsanity.com/paper/2205.06457/) (2022) - [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021) - [A Large-scale Dataset for Hate Speech Detection on Vietnamese Social Media Texts](https://mlsanity.com/paper/2103.11528/) (2021) - [IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization](https://mlsanity.com/paper/2109.04607/) (2021) - [Emotion Recognition for Vietnamese Social Media Text](https://mlsanity.com/paper/1911.09339/) (2019) - [Improving Sequence Tagging for Vietnamese Text Using Transformer-based Neural Models](https://mlsanity.com/paper/2006.15994/) (2020) - [RoBERTuito: a pre-trained language model for social media text in Spanish](https://mlsanity.com/paper/2111.09453/) (2021) - [PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing](https://mlsanity.com/paper/2101.01476/) (2021) - [ViHOS: Hate Speech Spans Detection for Vietnamese](https://mlsanity.com/paper/2301.10186/) (2023) - [ViDeBERTa: A powerful pre-trained language model for Vietnamese](https://mlsanity.com/paper/2301.10439/) (2023) - [Vietnamese Hate and Offensive Detection using PhoBERT-CNN and Social Media Streaming Data](https://mlsanity.com/paper/2206.00524/) (2022) - [SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamese](https://mlsanity.com/paper/2209.10482/) (2022) - [Exploiting Vietnamese Social Media Characteristics for Textual Emotion Recognition in Vietnamese](https://mlsanity.com/paper/2009.11005/) (2020) - [Detecting Spam Reviews on Vietnamese E-commerce Websites](https://mlsanity.com/paper/2207.14636/) (2022) ## Built on by (most cited first, 11 of 11) - [ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model](https://mlsanity.com/paper/2405.14141/) (2024) - [ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalization](https://mlsanity.com/paper/2501.07020/) (2025) - [A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Media](https://mlsanity.com/paper/2409.20467/) (2024) - [MTikGuard System: A Transformer-Based Multimodal System for Child-Safe Content Moderation on TikTok](https://mlsanity.com/paper/2511.17955/) (2025) - [ViTHSD: Exploiting Hatred by Targets for Hate Speech Detection on Vietnamese Social Media Texts](https://mlsanity.com/paper/2404.19252/) (2024) - [New Benchmark Dataset and Fine-Grained Cross-Modal Fusion Framework for Vietnamese Multimodal Aspect-Category Sentiment Analysis](https://mlsanity.com/paper/2405.00543/) (2024) - [ViToSA: Audio-Based Toxic Spans Detection on Vietnamese Speech Utterances](https://mlsanity.com/paper/2506.00636/) (2025) - [NEU-ESC: A Comprehensive Vietnamese dataset for Educational Sentiment analysis and topic Classification toward multitask learning](https://mlsanity.com/paper/2506.23524/) (2025) - [ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Texts](https://mlsanity.com/paper/2602.08371/) (2026) - [BamiBERT: A New BERT-based Language Model for Vietnamese](https://mlsanity.com/paper/2607.02259/) (2026) - [ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Texts](https://mlsanity.com/paper/2608.12776/) (2026) ## Abstract (arXiv) English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre-trained language models are still limited to Vietnamese social media tasks. In this paper, we present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT, which is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese social media texts using XLM-R architecture. Moreover, we explored our pre-trained model on five important natural language downstream tasks on Vietnamese social media texts: emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks. Our ViSoBERT model is available only for research purposes.