mlsanityData to 7 Oct 2026

Combining Static and Contextualised Multilingual Embeddings

arXiv 2203.09326 ↗First posted 17 Mar 2022cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We release the static embeddings and the continued pre-training code. Unlike most previous work, our continued pre-training approach does not require parallel text.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 40 of its 66 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

40 of 66 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  12. 2017Word Translation Without Parallel Datacited by 480
28 more
  1. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  2. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  3. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  4. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  5. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  6. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  7. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  8. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  9. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  10. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  11. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 69
  12. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  13. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  14. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61
  15. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  16. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  17. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  18. 2019Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsingcited by 46
  19. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  20. 2020Are All Good Word Vector Spaces Isomorphic?cited by 36
  21. 2020A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERTcited by 27
  22. 2021Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddingscited by 19
  23. 2021Obtaining Better Static Word Embeddings Using Contextual Embedding Modelscited by 18
  24. 2019Context-Aware Cross-Lingual Mappingcited by 17
  25. 2020fugashi, a Tool for Tokenizing Japanese in Pythoncited by 17
  26. 2021Probing Multilingual Language Models for Discoursecited by 12
  27. 2021XeroAlign: Zero-Shot Cross-lingual Transformer Alignmentcited by 11
  28. 2021Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Miningcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 3; 2025: 2.

8 papers.

  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  2. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  3. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  4. 2022Do Multilingual Language Models Capture Differing Moral Norms?cited by 15
  5. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  6. 2025Rethinking Hybrid Retrieval: When Small Embeddings and LLM Re-ranking Beat Bigger Modelscited by 1
  7. 2023CUNI Submission to MRL 2023 Shared Task on Multi-lingual Multi-task Information Retrievalcited by 0
  8. 2025How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigmcited by 0