mlsanityData to 7 Oct 2026

IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation

arXiv 2104.08200 ↗First posted 16 Apr 2021cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Natural language generation (NLG) benchmarks provide an important avenue to measure progress and develop better NLG systems. Unfortunately, the lack of publicly available NLG benchmarks for low-resource languages poses a challenging barrier for building NLG systems that work well for languages with limited amounts of data. Here we introduce IndoNLG, the first benchmark to measure natural language generation (NLG) progress in three low-resource -- yet widely spoken -- languages of Indonesia: Indonesian, Javanese, and Sundanese. Altogether, these languages are spoken by more than 100 million native speakers, and hence constitute an important use case of NLG systems today. Concretely, IndoNLG covers six tasks: summarization, question answering, chit-chat, and three different pairs of machine translation (MT) tasks. We collate a clean pretraining corpus of Indonesian, Sundanese, and Javanese datasets, Indo4B-Plus, which is used to pretrain our models: IndoBART and IndoGPT. We show that IndoBART and IndoGPT achieve competitive performance on all tasks -- despite using only one-fifth the parameters of a larger multilingual model, mBART-LARGE (Liu et al., 2020). This finding emphasizes the importance of pretraining on closely related, local languages to achieve more efficient learning and faster inference for very low-resource languages like Javanese and Sundanese.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 42 of its 74 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

42 of 74 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
30 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2019CamemBERT: a Tasty French Language Modelcited by 368
  8. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  9. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  10. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  11. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  12. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  13. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  14. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  15. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  16. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  17. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  18. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  19. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  20. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 92
  21. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  22. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  23. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  24. 2020XPersona: Evaluating Multilingual Personalized Chatbotcited by 36
  25. 2021WangchanBERTa: Pretraining transformer-based Thai Language Modelscited by 25
  26. 2020Low Resource Neural Machine Translation: A Benchmark for Five African Languagescited by 15
  27. 2020Establishing Baselines for Text Classification in Low-Resource Languagescited by 14
  28. 2020Liputan6: A Large-scale Indonesian Dataset for Text Summarizationcited by 14
  29. 2018IndoSum: A New Benchmark Dataset for Indonesian Text Summarizationcited by 13
  30. 2020Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesiscited by 8

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 15; 2023: 14; 2024: 16; 2025: 9.

57 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  5. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  6. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  8. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  9. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  10. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  11. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  12. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
38 more
  1. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  2. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  3. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  4. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  5. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  6. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  7. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  8. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  9. 2022Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?cited by 24
  10. 2023Dolphin: A Challenging and Diverse Benchmark for Arabic NLGcited by 22
  11. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  12. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  13. 2024Cheetah: Natural Language Generation for 517 African Languagescited by 15
  14. 2022This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polishcited by 13
  15. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  16. 2024LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalizationcited by 10
  17. 2024CELLO: Causal Evaluation of Large Vision-Language Modelscited by 10
  18. 2024Komodo: A Linguistic Expedition into Indonesia's Regional Languagescited by 9
  19. 2025Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asiacited by 8
  20. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 7
  21. 2024How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?cited by 6
  22. 2021Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximationcited by 5
  23. 2022Every picture tells a story: Image-grounded controllable stylistic story generationcited by 5
  24. 2023IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languagescited by 5
  25. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
  26. 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 4
  27. 2023MC2: Towards Transparent and Culturally-Aware NLP for Minority Languages in Chinacited by 4
  28. 2024Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesiacited by 4
  29. 2024Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sensecited by 4
  30. 2022Towards Answering Open-ended Ethical Quandary Questionscited by 3
  31. 2023The Obscure Limitation of Modular Multilingual Language Modelscited by 3
  32. 2022NusaCrowd: A Call for Open and Reproducible NLP Research in Indonesian Languagescited by 2
  33. 2023IndoToD: A Multi-Domain Indonesian Benchmark For End-to-End Task-Oriented Dialogue Systemscited by 2
  34. 2025Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translationcited by 2
  35. 2025SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?cited by 2
  36. 2022How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modelingcited by 1
  37. 2025RusCode: Russian Cultural Code Benchmark for Text-to-Image Generationcited by 1
  38. 2025IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesiancited by 1

50 most cited of 57 All 57 (JSON)