mlsanityData to 14 Oct 2026

Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching

arXiv 2209.05869 ↗First posted 13 Sep 2022cs.CLcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Previous studies have proved that cross-lingual knowledge distillation can significantly improve the performance of pre-trained models for cross-lingual similarity matching tasks. However, the student model needs to be large in this operation. Otherwise, its performance will drop sharply, thus making it impractical to be deployed to memory-limited devices. To address this issue, we delve into cross-lingual knowledge distillation and propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model. In our framework, contrastive learning, bottleneck, and parameter recurrent strategies are combined to prevent performance from being compromised during the compression process. The experimental results demonstrate that our method can compress the size of XLM-R and MiniLM by more than 50%, while the performance is only reduced by about 1%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 32 of its 39 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

32 of 39 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
20 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  2. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  3. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  4. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  6. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  9. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  10. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  11. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  12. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  13. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  14. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  15. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 62
  16. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  17. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  18. 2018Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Modelcited by 46
  19. 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
  20. 2021Lightweight Cross-Lingual Sentence Representation Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Modeling Fine-grained Information via Knowledge-aware Hierarchical Graph for Zero-shot Entity Retrievalcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
  2. 2025Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching
  3. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation
  4. 2024Dual-Space Knowledge Distillation for Large Language Models
  5. 2025Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual
  6. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  7. 2025AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages
  8. 2023A Cohesive Distillation Architecture for Neural Language Models