# Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning

arXiv: https://arxiv.org/abs/2309.08185 · HTML: https://mlsanity.com/paper/2309.08185/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [SGPT: GPT Sentence Embeddings for Semantic Search](https://mlsanity.com/paper/2202.08904/) (2022)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [learn2learn: A Library for Meta-Learning Research](https://mlsanity.com/paper/2008.12284/) (2020)
- [Zero-Shot Cross-Lingual Transfer with Meta Learning](https://mlsanity.com/paper/2003.02739/) (2020)
- [BERT Learns to Teach: Knowledge Distillation with Meta Learning](https://mlsanity.com/paper/2106.04570/) (2021)
- [LAReQA: Language-agnostic answer retrieval from a multilingual pool](https://mlsanity.com/paper/2004.05484/) (2020)
- [Meta Learning for End-to-End Low-Resource Speech Recognition](https://mlsanity.com/paper/1910.12094/) (2019)
- [Semi-supervised Question Retrieval with Gated Convolutions](https://mlsanity.com/paper/1512.05726/) (2015)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Gradual Fine-Tuning for Low-Resource Domain Adaptation](https://mlsanity.com/paper/2103.02205/) (2021)
- [Meta-Transfer Learning for Code-Switched Speech Recognition](https://mlsanity.com/paper/2004.14228/) (2020)
- [On Cross-Lingual Retrieval with Multilingual Text Encoders](https://mlsanity.com/paper/2112.11031/) (2021)
- [Meta Knowledge Distillation](https://mlsanity.com/paper/2202.07940/) (2022)

## Built on by (most cited first, 1 of 1)

- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)

## Abstract (arXiv)

Multilingual semantic search is the task of retrieving relevant contents to a query expressed in different language combinations. This requires a better semantic understanding of the user's intent and its contextual meaning. Multilingual semantic search is less explored and more challenging than its monolingual or bilingual counterparts, due to the lack of multilingual parallel resources for this task and the need to circumvent "language bias". In this work, we propose an alignment approach: MAML-Align, specifically for low-resource scenarios. Our approach leverages meta-distillation learning based on MAML, an optimization-based Model-Agnostic Meta-Learner. MAML-Align distills knowledge from a Teacher meta-transfer model T-MAML, specialized in transferring from monolingual to bilingual semantic search, to a Student model S-MAML, which meta-transfers from bilingual to multilingual semantic search. To the best of our knowledge, we are the first to extend meta-distillation to a multilingual search application. Our empirical results show that on top of a strong baseline based on sentence transformers, our meta-distillation approach boosts the gains provided by MAML and significantly outperforms naive fine-tuning methods. Furthermore, multilingual meta-distillation learning improves generalization even to unseen languages.