mlsanityData to 7 Oct 2026

LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation

arXiv 2302.08387 ↗First posted 16 Feb 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large-scale language-agnostic sentence embedding models such as LaBSE (Feng et al., 2022) obtain state-of-the-art performance for parallel sentence alignment. However, these large-scale models can suffer from inference speed and computation overhead. This study systematically explores learning language-agnostic sentence embeddings with lightweight models. We demonstrate that a thin-deep encoder can construct robust low-dimensional sentence embeddings for 109 languages. With our proposed distillation methods, we achieve further improvements by incorporating knowledge from a teacher model. Empirical results on Tatoeba, United Nations, and BUCC show the effectiveness of our lightweight models. We release our lightweight language-agnostic sentence embedding models LEALLA on TensorFlow Hub.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

15 of 23 references.

References · 15

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  6. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  9. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  10. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  11. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  12. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
3 more
  1. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  2. 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
  3. 2021Lightweight Cross-Lingual Sentence Representation Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 1; 2026: 1.

11 papers.

  1. 2023Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embeddingcited by 9
  2. 2023Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularizationcited by 7
  3. 2024Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Datacited by 7
  4. 2024BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Languagecited by 6
  5. 2024Presence or Absence: Are Unknown Word Usages in Dictionaries?cited by 3
  6. 2024AXOLOTL'24 Shared Task on Multilingual Explainable Semantic Change Modelingcited by 3
  7. 2025BiMax: Bidirectional MaxSim Score for Document-Level Alignmentcited by 2
  8. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0
  9. 2024Making Sentence Embeddings Robust to User-Generated Contentcited by 0
  10. 2024A Disease Labeler for Chinese Chest X-Ray Report Generationcited by 0
  11. 2026Bootstrapping Embeddings for Low Resource Languagescited by 0