mlsanityData to 7 Oct 2026

Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes

arXiv 2211.05015 ↗First posted 9 Nov 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Providing better language tools for low-resource and endangered languages is imperative for equitable growth. Recent progress with massively multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages. However, this transfer is not universal, with many languages not currently understood by multilingual approaches. It is estimated that only 72 languages possess a "small set of labeled datasets" on which we could test a model's performance, the vast majority of languages not having the resources available to simply evaluate performances on. In this work, we attempt to clarify which languages do and do not currently benefit from such transfer. To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language. We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

37 of 51 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2019How multilingual is Multilingual BERT?cited by 642
25 more
  1. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  2. 2019CamemBERT: a Tasty French Language Modelcited by 368
  3. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  4. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  5. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  6. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  7. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  8. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
  9. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  10. 2019BERTje: A Dutch BERT Modelcited by 109
  11. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  12. 2020UnNatural Language Inferencecited by 68
  13. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  14. 2021What Context Features Can Transformer Language Models Use?cited by 49
  15. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  16. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
  17. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  18. 2020PTT5: Pretraining and validating the T5 model on Brazilian Portuguese datacited by 35
  19. 2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
  20. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  21. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  22. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  23. 2021Analysing The Impact Of Linguistic Features On Cross-Lingual Transfercited by 16
  24. 2021Local Structure Matters Most: Perturbation Study in NLUcited by 10
  25. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuningcited by 3
  2. 2024UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understandingcited by 0