# Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

arXiv: https://arxiv.org/abs/2201.08471 · HTML: https://mlsanity.com/paper/2201.08471/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 20 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [CEDR: Contextualized Embeddings for Document Ranking](https://mlsanity.com/paper/1904.07094/) (2019)
- [Simple Applications of BERT for Ad Hoc Document Retrieval](https://mlsanity.com/paper/1903.10972/) (2019)
- [Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval](https://mlsanity.com/paper/2106.11251/) (2021)
- [Cross-lingual Information Retrieval with BERT](https://mlsanity.com/paper/2004.13005/) (2020)
- [PACRR: A Position-Aware Neural IR Model for Relevance Matching](https://mlsanity.com/paper/1704.03940/) (2017)
- [Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning](https://mlsanity.com/paper/1912.13080/) (2019)
- [The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020](https://mlsanity.com/paper/2008.04885/) (2020)
- [A Study of Neural Matching Models for Cross-lingual IR](https://mlsanity.com/paper/2005.12994/) (2020)
- [Improving Low-Resource Cross-lingual Document Retrieval by Reranking with Deep Bilingual Representations](https://mlsanity.com/paper/1906.03492/) (2019)
- [Cross-Lingual Relevance Transfer for Document Retrieval](https://mlsanity.com/paper/1911.02989/) (2019)
- [Cross-Lingual Training with Dense Retrieval for Document Retrieval](https://mlsanity.com/paper/2109.01628/) (2021)

## Built on by (most cited first, 30 of 36)

- [WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia](https://mlsanity.com/paper/2305.14292/) (2023)
- [Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages](https://mlsanity.com/paper/2210.09984/) (2022)
- [Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever](https://mlsanity.com/paper/2408.16672/) (2024)
- [Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation](https://mlsanity.com/paper/2401.04810/) (2024)
- [Neural Approaches to Multilingual Information Retrieval](https://mlsanity.com/paper/2209.01335/) (2022)
- [Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval](https://mlsanity.com/paper/2204.02292/) (2022)
- [Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation](https://mlsanity.com/paper/2301.12566/) (2023)
- [C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval](https://mlsanity.com/paper/2204.11989/) (2022)
- [HC4: A New Suite of Test Collections for Ad Hoc CLIR](https://mlsanity.com/paper/2201.09992/) (2022)
- [Soft Prompt Decoding for Multilingual Dense Retrieval](https://mlsanity.com/paper/2305.09025/) (2023)
- [Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector](https://mlsanity.com/paper/2510.00671/) (2025)
- [MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion](https://mlsanity.com/paper/2503.20698/) (2025)
- [Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models](https://mlsanity.com/paper/2407.05502/) (2024)
- [mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval](https://mlsanity.com/paper/2501.19264/) (2025)
- [An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback](https://mlsanity.com/paper/2311.11226/) (2023)
- [An Interdisciplinary Approach to Human-Centered Machine Translation](https://mlsanity.com/paper/2506.13468/) (2025)
- [Query Decomposition for RAG: Balancing Exploration-Exploitation](https://mlsanity.com/paper/2510.18633/) (2025)
- [Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters](https://mlsanity.com/paper/2212.10448/) (2022)
- [An Experimental Study on Pretraining Transformers from Scratch for IR](https://mlsanity.com/paper/2301.10444/) (2023)
- [Distillation for Multilingual Information Retrieval](https://mlsanity.com/paper/2405.00977/) (2024)
- [V-Agent: An Interactive Video Search System Using Vision-Language Models](https://mlsanity.com/paper/2512.16925/) (2025)
- [CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents](https://mlsanity.com/paper/2504.16264/) (2025)
- [Simple Projection Variants Improve ColBERT Performance](https://mlsanity.com/paper/2510.12327/) (2025)
- [HLTCOE at TREC 2023 NeuCLIR Track](https://mlsanity.com/paper/2404.08118/) (2024)
- [Extending Translate-Train for ColBERT-X to African Language CLIR](https://mlsanity.com/paper/2404.08134/) (2024)
- [Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios](https://mlsanity.com/paper/2509.15380/) (2025)
- [LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026](https://mlsanity.com/paper/2511.00444/) (2025)
- [A Comparative Study of Text Retrieval Models on DaReCzech](https://mlsanity.com/paper/2411.12921/) (2024)
- [Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO](https://mlsanity.com/paper/2412.12997/) (2024)
- [How Vital is the Jurisprudential Relevance: Law Article Intervened Legal Case Retrieval and Matching](https://mlsanity.com/paper/2502.18292/) (2025)

## Abstract (arXiv)

The advent of transformer-based models such as BERT has led to the rise of neural ranking models. These models have improved the effectiveness of retrieval systems well beyond that of lexical term matching models such as BM25. While monolingual retrieval tasks have benefited from large-scale training collections such as MS MARCO and advances in neural architectures, cross-language retrieval tasks have fallen behind these advancements. This paper introduces ColBERT-X, a generalization of the ColBERT multi-representation dense retrieval model that uses the XLM-RoBERTa (XLM-R) encoder to support cross-language information retrieval (CLIR). ColBERT-X can be trained in two ways. In zero-shot training, the system is trained on the English MS MARCO collection, relying on the XLM-R encoder for cross-language mappings. In translate-train, the system is trained on the MS MARCO English queries coupled with machine translations of the associated MS MARCO passages. Results on ad hoc document ranking tasks in several languages demonstrate substantial and statistically significant improvements of these trained dense retrieval models over traditional lexical CLIR baselines.