mlsanityData to 14 Oct 2026

NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

arXiv 2403.01817 ↗First posted 4 Mar 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Indonesia's linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world's most linguistically rich nations. This diversity, coupled with the widespread practice of code-switching and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages and dialects. Through rigorous evaluation across a range of benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks involving multiple languages of Indonesia, paving the way for future natural language understanding research for under-represented languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

26 of 43 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  12. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
14 more
  1. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  2. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  3. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  4. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  5. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  6. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  7. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 58
  8. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  9. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  10. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  11. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  12. 2021WangchanBERTa: Pretraining transformer-based Thai Language Modelscited by 26
  13. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  14. 2023IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languagescited by 5

Built on by

Not cited yet.