# Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

arXiv: https://arxiv.org/abs/2401.04810 · HTML: https://mlsanity.com/paper/2401.04810/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Deeper Text Understanding for IR with Contextual Neural Language Modeling](https://mlsanity.com/paper/1905.09217/) (2019)
- [PLAID: An Efficient Engine for Late Interaction Retrieval](https://mlsanity.com/paper/2205.09707/) (2022)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment](https://mlsanity.com/paper/2106.06381/) (2021)
- [Curriculum Learning for Dense Retrieval Distillation](https://mlsanity.com/paper/2204.13679/) (2022)
- [Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models](https://mlsanity.com/paper/2201.08471/) (2022)
- [Simplified Data Wrangling with ir_datasets](https://mlsanity.com/paper/2103.02280/) (2021)
- [PROD: Progressive Distillation for Dense Retrieval](https://mlsanity.com/paper/2209.13335/) (2022)
- [Learning Cross-Lingual IR from an English Retriever](https://mlsanity.com/paper/2112.08185/) (2021)
- [C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval](https://mlsanity.com/paper/2204.11989/) (2022)
- [Overview of the TREC 2022 NeuCLIR Track](https://mlsanity.com/paper/2304.12367/) (2023)

## Built on by (most cited first, 22 of 22)

- [jina-embeddings-v5-text: Task-Targeted Embedding Distillation](https://mlsanity.com/paper/2602.15547/) (2026)
- [CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation](https://mlsanity.com/paper/2406.17186/) (2024)
- [Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector](https://mlsanity.com/paper/2510.00671/) (2025)
- [MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion](https://mlsanity.com/paper/2503.20698/) (2025)
- [Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models](https://mlsanity.com/paper/2407.05502/) (2024)
- [Query Decomposition for RAG: Balancing Exploration-Exploitation](https://mlsanity.com/paper/2510.18633/) (2025)
- [Overview of the TREC 2025 RAGTIME Track](https://mlsanity.com/paper/2602.10024/) (2026)
- [Distillation for Multilingual Information Retrieval](https://mlsanity.com/paper/2405.00977/) (2024)
- [Quality-Aware Translation Tagging in Multilingual RAG system](https://mlsanity.com/paper/2510.23070/) (2025)
- [CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training](https://mlsanity.com/paper/2604.05821/) (2026)
- [CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents](https://mlsanity.com/paper/2504.16264/) (2025)
- [Incorporating Q&A Nuggets into Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.13222/) (2026)
- [HLTCOE at TREC 2023 NeuCLIR Track](https://mlsanity.com/paper/2404.08118/) (2024)
- [RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.10644/) (2026)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage](https://mlsanity.com/paper/2603.08819/) (2026)
- [DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation](https://mlsanity.com/paper/2605.04458/) (2026)
- [A Comparative Study of Text Retrieval Models on DaReCzech](https://mlsanity.com/paper/2411.12921/) (2024)
- [MURR: Model Updating with Regularized Replay for Searching a Document Stream](https://mlsanity.com/paper/2504.10250/) (2025)
- [MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries](https://mlsanity.com/paper/2505.16631/) (2025)
- [ColBERTSaR: Sparsified ColBERT Index via Product Quantization](https://mlsanity.com/paper/2606.05568/) (2026)
- [Dual-Stream Simultaneous Translation via 2D Grid Attention](https://mlsanity.com/paper/2609.34902/) (2026)

## Abstract (arXiv)

Prior work on English monolingual retrieval has shown that a cross-encoder trained using a large number of relevance judgments for query-document pairs can be used as a teacher to train more efficient, but similarly effective, dual-encoder student models. Applying a similar knowledge distillation approach to training an efficient dual-encoder model for Cross-Language Information Retrieval (CLIR), where queries and documents are in different languages, is challenging due to the lack of a sufficiently large training collection when the query and document languages differ. The state of the art for CLIR thus relies on translating queries, documents, or both from the large English MS MARCO training set, an approach called Translate-Train. This paper proposes an alternative, Translate-Distill, in which knowledge distillation from either a monolingual cross-encoder or a CLIR cross-encoder is used to train a dual-encoder CLIR student model. This richer design space enables the teacher model to perform inference in an optimized setting, while training the student model directly for CLIR. Trained models and artifacts are publicly available on Huggingface.