mlsanityData to 7 Oct 2026

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning

arXiv 2210.09934 ↗First posted 18 Oct 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embeddings without semantic loss, thereby improving cross-lingual transferability. Experimental results on mBERT and XLM-R demonstrate that our method significantly outperforms previous works on the zero-shot cross-lingual text classification task and can obtain a better multilingual alignment.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

30 of 36 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2017Graph Attention Networkscited by 8,854
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
18 more
  1. 2018Generating Natural Language Adversarial Examplescited by 422
  2. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  3. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  4. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  5. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  6. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  7. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  8. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  9. 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
  10. 2020On Learning Universal Representations Across Languagescited by 58
  11. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  12. 2020GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extractioncited by 43
  13. 2019Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systemscited by 42
  14. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 41
  15. 2020Multilingual BERT Post-Pretraining Alignmentcited by 30
  16. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  17. 2019Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsingcited by 11
  18. 2021Syntax-augmented Multilingual BERT for Cross-lingual Transfercited by 11

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  2. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 22
  3. 2023Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuningcited by 5
  4. 2024Zero-shot Cross-lingual Stance Detection via Adversarial Language Adaptationcited by 1