# Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

arXiv: https://arxiv.org/abs/2301.12566 · HTML: https://mlsanity.com/paper/2301.12566/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 29 Jan 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2010.02666/) (2020)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [Improving GANs Using Optimal Transport](https://mlsanity.com/paper/1803.05573/) (2018)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [Knowledge Distillation via the Target-aware Transformer](https://mlsanity.com/paper/2205.10793/) (2022)
- [Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation](https://mlsanity.com/paper/2112.10063/) (2021)
- [Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation](https://mlsanity.com/paper/2112.09445/) (2021)
- [Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models](https://mlsanity.com/paper/2201.08471/) (2022)
- [Improving Sequence-to-Sequence Learning via Optimal Transport](https://mlsanity.com/paper/1901.06283/) (2019)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [Learning Cross-Lingual IR from an English Retriever](https://mlsanity.com/paper/2112.08185/) (2021)
- [A Study of Neural Matching Models for Cross-lingual IR](https://mlsanity.com/paper/2005.12994/) (2020)
- [Unsupervised Cross-Lingual Information Retrieval using Monolingual Data Only](https://mlsanity.com/paper/1805.00879/) (2018)

## Built on by (most cited first, 15 of 15)

- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [Soft Prompt Decoding for Multilingual Dense Retrieval](https://mlsanity.com/paper/2305.09025/) (2023)
- [Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I](https://mlsanity.com/paper/2407.02464/) (2024)
- [Variance reduction in output from generative AI](https://mlsanity.com/paper/2503.01033/) (2025)
- [IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languages](https://mlsanity.com/paper/2312.09508/) (2023)
- [Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data](https://mlsanity.com/paper/2305.05295/) (2023)
- [Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide](https://mlsanity.com/paper/2411.09539/) (2024)
- [OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature](https://mlsanity.com/paper/2505.22945/) (2025)
- [CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training](https://mlsanity.com/paper/2604.05821/) (2026)
- [Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2307.04096/) (2023)
- [CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents](https://mlsanity.com/paper/2504.16264/) (2025)
- [Improving Cross-lingual Representation for Semantic Retrieval with Code-switching](https://mlsanity.com/paper/2403.01364/) (2024)
- [Many-to-Many Matching via Sparsity Controlled Optimal Transport](https://mlsanity.com/paper/2503.24204/) (2025)
- [MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries](https://mlsanity.com/paper/2505.16631/) (2025)
- [SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali](https://mlsanity.com/paper/2603.29221/) (2026)

## Abstract (arXiv)

Benefiting from transformer-based pre-trained language models, neural ranking models have made significant progress. More recently, the advent of multilingual pre-trained language models provides great support for designing neural cross-lingual retrieval models. However, due to unbalanced pre-training data in different languages, multilingual language models have already shown a performance gap between high and low-resource languages in many downstream tasks. And cross-lingual retrieval models built on such pre-trained models can inherit language bias, leading to suboptimal result for low-resource languages. Moreover, unlike the English-to-English retrieval task, where large-scale training collections for document ranking such as MS MARCO are available, the lack of cross-lingual retrieval data for low-resource language makes it more challenging for training cross-lingual retrieval models. In this work, we propose OPTICAL: Optimal Transport distillation for low-resource Cross-lingual information retrieval. To transfer a model from high to low resource languages, OPTICAL forms the cross-lingual token alignment task as an optimal transport problem to learn from a well-trained monolingual retrieval model. By separating the cross-lingual knowledge from knowledge of query document matching, OPTICAL only needs bitext data for distillation training, which is more feasible for low-resource languages. Experimental results show that, with minimal training data, OPTICAL significantly outperforms strong baselines on low-resource languages, including neural machine translation.