mlsanityData to 7 Oct 2026

AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages

arXiv 2104.08726 ↗First posted 18 Apr 2021cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis-only models. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%. Continued pretraining offers improvements, with an average accuracy of 44.05%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 48.72%.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 37 of its 96 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

37 of 96 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,578
25 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  5. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  6. 2019How multilingual is Multilingual BERT?cited by 642
  7. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  8. 2017Word Translation Without Parallel Datacited by 480
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  10. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  11. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  13. 2018Learning Word Vectors for 157 Languagescited by 279
  14. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  15. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  16. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  17. 2020With Little Power Comes Great Responsibilitycited by 138
  18. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  19. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  20. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  21. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  22. 2018Challenges of language technologies for the indigenous languages of the Americascited by 29
  23. 2016Massively Multilingual Word Embeddingscited by 26
  24. 2020FarsTail: A Persian Natural Language Inference Datasetcited by 23
  25. 2019Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Setcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 15; 2023: 19; 2024: 5; 2025: 6; 2026: 1.

50 papers.

Sort:

By area:

  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  2. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  3. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  4. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  5. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  6. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  7. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  8. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  9. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  10. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  11. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  12. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
38 more
  1. 2024Zero-Shot Tokenizer Transfercited by 30
  2. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
  3. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  4. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  5. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  6. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  7. 2024From One to Many: Expanding the Scope of Toxicity Mitigation in Language Modelscited by 18
  8. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  9. 2023Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systemscited by 16
  10. 2022BERTifying Sinhala -- A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classificationcited by 15
  11. 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
  12. 2023Translation and Fusion Improves Zero-shot Cross-lingual Information Extractioncited by 15
  13. 2023Cross-Lingual Transfer with Target Language-Ready Task Adapterscited by 15
  14. 2022JamPatoisNLI: A Jamaican Patois Natural Language Inference Datasetcited by 11
  15. 2023CreoleVal: Multilingual Multitask Benchmarks for Creolescited by 9
  16. 2023Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averagingcited by 8
  17. 2023Neural Machine Translation for the Indigenous Languages of the Americas: An Introductioncited by 7
  18. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  19. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  20. 2022The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfercited by 5
  21. 2025Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Taskscited by 5
  22. 2023Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Modelscited by 4
  23. 2023Enhancing Translation for Indigenous Languages: Experiments with Multilingual Modelscited by 4
  24. 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 4
  25. 2022InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examplescited by 3
  26. 2024NLP Progress in Indigenous Latin American Languagescited by 3
  27. 2023Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognitioncited by 2
  28. 2025La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin Americacited by 2
  29. 2021Multilingual unsupervised sequence segmentation transfers to extremely low-resource languagescited by 1
  30. 2023DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfercited by 1
  31. 2023EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learningcited by 1
  32. 2022Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual modelscited by 0
  33. 2023tmn at SemEval-2023 Task 9: Multilingual Tweet Intimacy Detection using XLM-T, Google Translate, and Ensemble Learningcited by 0
  34. 2024Effectively Prompting Small-sized Language Models for Cross-lingual Tasks via Winning Ticketscited by 0
  35. 2025Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval modelscited by 0
  36. 2025CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languagescited by 0
  37. 2025Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languagescited by 0
  38. 2026The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraintscited by 0