mlsanityData to 14 Oct 2026

CreoleVal: Multilingual Multitask Benchmarks for Creoles

arXiv 2310.19567 ↗First posted 30 Oct 2023cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research.While the genealogical ties between Creoles and a number of highly-resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning 8 different NLP tasks, covering up to 28 Creole languages; it is an aggregate of novel development datasets for reading comprehension, relation classification, and machine translation for Creoles, in addition to a practical gateway to a handful of preexisting benchmarks. For each benchmark, we conduct baseline experiments in a zero-shot setting in order to further ascertain the capabilities and limitations of transfer learning for Creoles. Ultimately, we see CreoleVal as an opportunity to empower research on Creoles in NLP and computational linguistics, and in general, a step towards more equitable language technology around the globe.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 61 of its 141 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

61 of 141 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
38 more
  1. 2019How multilingual is Multilingual BERT?cited by 645
  2. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  3. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  4. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  5. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  6. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  7. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  8. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  9. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  10. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  11. 2020Participation is not a Design Fix for Machine Learningcited by 111
  12. 2018Unsupervised Alignment of Embeddings with Wasserstein Procrustescited by 97
  13. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  14. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  15. 2022NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysiscited by 64
  16. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  17. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  18. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  19. 2021ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learningcited by 46
  20. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  21. 2016Question Answering on Freebase via Relation Extraction and Textual Evidencecited by 39
  22. 2020SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflectioncited by 39
  23. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  24. 2022What a Creole Wants, What a Creole Needscited by 27
  25. 2023Ethical Considerations for Machine Translation of Indigenous Languages: Giving a Voice to the Speakerscited by 27
  26. 2021On Language Models for Creolescited by 17
  27. 2022Factual Consistency of Multilingual Pretrained Language Modelscited by 15
  28. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  29. 2019OpenTapioca: Lightweight Entity Linking for Wikidatacited by 14
  30. 2022Language-Family Adapters for Low-Resource Multilingual Neural Machine Translationcited by 14
  31. 2020SIGTYP 2020 Shared Task: Prediction of Typological Featurescited by 13
  32. 2020The RELX Dataset and Matching the Multilingual Blanks for Cross-Lingual Relation Classificationcited by 13
  33. 2020Semantic Enrichment of Nigerian Pidgin English for Contextual Sentiment Classificationcited by 12
  34. 2021Adapting Monolingual Models: Data can be Scarce when Language Similarity is Highcited by 12
  35. 2021To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLPcited by 12
  36. 2014Cross-lingual Annotation Projection for Semantic Rolescited by 11
  37. 2019Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translationcited by 11
  38. 2019PidginUNMT: Unsupervised Neural Machine Translation from West African Pidgin to Englishcited by 11

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 5.

9 papers.

  1. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  2. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  3. 2024Krey`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languagescited by 10
  4. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  5. 2024BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of Englishcited by 5
  6. 2025Large Multimodal Models for Low-Resource Languages: A Surveycited by 4
  7. 2025Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participationcited by 3
  8. 2025Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creolecited by 2
  9. 2025Kr'eyoLID From Language Identification Towards Language Miningcited by 0