mlsanityData to 14 Oct 2026

IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization

arXiv 2109.04607 ↗First posted 10 Sep 2021cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that is trained by extending a monolingually-trained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

21 of 38 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  10. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  11. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  12. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
9 more
  1. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  2. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  3. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  4. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 58
  5. 2020Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QAcited by 23
  6. 2018Parsing Tweets into Universal Dependenciescited by 19
  7. 2020Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vectorcited by 18
  8. 2020Liputan6: A Large-scale Indonesian Dataset for Text Summarizationcited by 14
  9. 2020Parsing with Multilingual BERT, a Small Corpus, and a Small Treebankcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 6; 2025: 7; 2026: 1.

25 papers.

Sort:

By area:

  1. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  2. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  3. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  4. 2022Language Modelling with Pixelscited by 55
  5. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  6. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  7. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  8. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  9. 2023ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processingcited by 11
  10. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  11. 2024Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexiconcited by 7
  12. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
13 more
  1. 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 5
  2. 2024Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesiacited by 4
  3. 2025Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Settingcited by 4
  4. 2022NusaCrowd: A Call for Open and Reproducible NLP Research in Indonesian Languagescited by 2
  5. 2025Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptationcited by 2
  6. 2022Location-based Twitter Filtering for the Creation of Low-Resource Language Datasets in Indonesian Local Languagescited by 1
  7. 2025SafeSpeech: A Comprehensive and Interactive Tool for Analysing Sexist and Abusive Language in Conversationscited by 1
  8. 2023RoBERTweet: A BERT Language Model for Romanian Tweetscited by 0
  9. 2024NusaBERT: Teaching IndoBERT to be Multilingual and Multiculturalcited by 0
  10. 2025A Multi-Labeled Dataset for Indonesian Discourse: Examining Toxicity, Polarization, and Demographics Informationcited by 0
  11. 2025Dual-Class Prompt Generation: Enhancing Indonesian Gender-Based Hate Speech Detection through Data Augmentationcited by 0
  12. 2025A Gamified Evaluation and Recruitment Platform for Low Resource Language Machine Translation Systemscited by 0
  13. 2026A Comparative Study of PyCaret AutoML and CNN-BiLSTM for Binary Hate Speech Detection in Indonesian Twittercited by 0