mlsanityData to 7 Oct 2026

Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation

arXiv 2109.00194 ↗First posted 1 Sep 2021cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent multilingual pre-trained language models have achieved remarkable zero-shot performance, where the model is only finetuned on one source language and directly evaluated on target languages. In this work, we propose a self-learning framework that further utilizes unlabeled data of target languages, combined with uncertainty estimation in the process to select high-quality silver labels. Three different uncertainties are adapted and analyzed specifically for the cross lingual transfer: Language Heteroscedastic/Homoscedastic Uncertainty (LEU/LOU), Evidential Uncertainty (EVI). We evaluate our framework with uncertainties on two cross-lingual tasks including Named Entity Recognition (NER) and Natural Language Inference (NLI) covering 40 languages in total, which outperforms the baselines significantly by 10 F1 on average for NER and 2.5 accuracy score for NLI.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

15 of 22 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  11. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  12. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
3 more
  1. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  2. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
  3. 2019Mitigating Uncertainty in Document Classificationcited by 22

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1.

4 papers.

  1. 2022A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learningcited by 6
  2. 2022SEED: Sound Event Early Detection via Evidential Uncertaintycited by 4
  3. 2021Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-sentence Dependency Graphcited by 2
  4. 2023Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Setscited by 2