mlsanityData to 7 Oct 2026

ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing

arXiv 2310.11166 ↗First posted 17 Oct 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre-trained language models are still limited to Vietnamese social media tasks. In this paper, we present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT, which is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese social media texts using XLM-R architecture. Moreover, we explored our pre-trained model on five important natural language downstream tasks on Vietnamese social media texts: emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks. Our ViSoBERT model is available only for research purposes.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

29 of 41 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  11. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  12. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
17 more
  1. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  2. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  3. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 44
  4. 2022ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generationcited by 37
  5. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  6. 2021A Large-scale Dataset for Hate Speech Detection on Vietnamese Social Media Textscited by 25
  7. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  8. 2019Emotion Recognition for Vietnamese Social Media Textcited by 23
  9. 2020Improving Sequence Tagging for Vietnamese Text Using Transformer-based Neural Modelscited by 22
  10. 2021RoBERTuito: a pre-trained language model for social media text in Spanishcited by 22
  11. 2021PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsingcited by 13
  12. 2023ViHOS: Hate Speech Spans Detection for Vietnamesecited by 13
  13. 2023ViDeBERTa: A powerful pre-trained language model for Vietnamesecited by 11
  14. 2022Vietnamese Hate and Offensive Detection using PhoBERT-CNN and Social Media Streaming Datacited by 9
  15. 2022SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamesecited by 7
  16. 2020Exploiting Vietnamese Social Media Characteristics for Textual Emotion Recognition in Vietnamesecited by 6
  17. 2022Detecting Spam Reviews on Vietnamese E-commerce Websitescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Modelcited by 3
  2. 2025ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalizationcited by 2
  3. 2024A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Mediacited by 1
  4. 2024ViTHSD: Exploiting Hatred by Targets for Hate Speech Detection on Vietnamese Social Media Textscited by 0
  5. 2024New Benchmark Dataset and Fine-Grained Cross-Modal Fusion Framework for Vietnamese Multimodal Aspect-Category Sentiment Analysiscited by 0
  6. 2025ViToSA: Audio-Based Toxic Spans Detection on Vietnamese Speech Utterancescited by 0
  7. 2025NEU-ESC: A Comprehensive Vietnamese dataset for Educational Sentiment analysis and topic Classification toward multitask learningcited by 0
  8. 2025MTikGuard System: A Transformer-Based Multimodal System for Child-Safe Content Moderation on TikTokcited by 0
  9. 2026ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Textscited by 0
  10. 2026BamiBERT: A New BERT-based Language Model for Vietnamesecited by 0
  11. 2026ViTOED: A Dataset for Target-Oriented Emotion Detection on Vietnamese Social Media Textscited by 0