mlsanityData to 7 Oct 2026

Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders

arXiv 2205.00267 ↗First posted 30 Apr 2022cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, it remains unclear how to best leverage them to represent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a simple yet efficient method for exposing the cross-lingual lexical knowledge by means of additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. Using bilingual lexical induction (BLI), cross-lingual lexical semantic similarity, and cross-lingual entity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LaBSE can be 'rewired' into effective cross-lingual lexical encoders via the contrastive learning procedure, and that they contain more cross-lingual lexical knowledge than what 'meets the eye' when they are used as off-the-shelf SEs. This way, we also provide an effective tool for harnessing 'covert' multilingual lexical knowledge hidden in multilingual sentence encoders.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 30 Apr 2022.

Builds on

45 of 59 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
33 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  2. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  3. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  4. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  5. 2017Word Translation Without Parallel Datacited by 480
  6. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  7. 2020A Metric Learning Reality Checkcited by 317
  8. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  9. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  10. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  11. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  12. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  13. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  14. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  15. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  16. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  17. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  18. 2019How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptionscited by 48
  19. 2019Training Neural Response Selection for Task-Oriented Dialogue Systemscited by 48
  20. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  21. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 41
  22. 2020Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similaritycited by 38
  23. 2019Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?cited by 31
  24. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  25. 2021Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linkingcited by 27
  26. 2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
  27. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  28. 2021DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddingscited by 19
  29. 2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
  30. 2019What do Language Representations Really Represent?cited by 17
  31. 2022Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structurecited by 17
  32. 2019Semantic Drift in Multilingual Representationscited by 13
  33. 2021Prix-LM: Pretraining for Multilingual Knowledge Base Constructioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 1.

7 papers.

  1. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  2. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluationcited by 4
  3. 2024Self-Augmented In-Context Learning for Unsupervised Word Translationcited by 4
  4. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
  5. 2025Semantic Pivots Enable Cross-Lingual Transfer in Large Language Modelscited by 1
  6. 2024A survey of neural-network-based methods utilising comparable data for finding translation equivalentscited by 0
  7. 2026To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarioscited by 0