# On Cross-Lingual Retrieval with Multilingual Text Encoders

arXiv: https://arxiv.org/abs/2112.11031 · HTML: https://mlsanity.com/paper/2112.11031/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 89 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)

## Built on by (most cited first, 22 of 22)

- [XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2210.13693/) (2022)
- [DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine](https://mlsanity.com/paper/2203.10232/) (2022)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval](https://mlsanity.com/paper/2204.11989/) (2022)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Skill Extraction from Job Postings using Weak Supervision](https://mlsanity.com/paper/2209.08071/) (2022)
- [Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders](https://mlsanity.com/paper/2205.00267/) (2022)
- [Evaluating Large Language Models for Cross-Lingual Retrieval](https://mlsanity.com/paper/2509.14749/) (2025)
- [Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters](https://mlsanity.com/paper/2212.10448/) (2022)
- [Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data](https://mlsanity.com/paper/2305.05295/) (2023)
- [Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers](https://mlsanity.com/paper/2310.12575/) (2023)
- [ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System](https://mlsanity.com/paper/2205.14981/) (2022)
- [Improving Domain-Specific Retrieval by NLI Fine-Tuning](https://mlsanity.com/paper/2308.03103/) (2023)
- [Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning](https://mlsanity.com/paper/2309.08185/) (2023)
- [What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models](https://mlsanity.com/paper/2511.19324/) (2025)
- [BiST: A Gold Standard Bangla-English Bilingual Corpus for Sentence Structure and Tense Classification with Inter-Annotator Agreement](https://mlsanity.com/paper/2604.04708/) (2026)
- [A General-Purpose Multilingual Document Encoder](https://mlsanity.com/paper/2305.07016/) (2023)
- [Mapping Transformer Leveraged Embeddings for Cross-Lingual Document Representation](https://mlsanity.com/paper/2401.06583/) (2024)
- [Domain Adaptation of Multilingual Semantic Search -- Literature Review](https://mlsanity.com/paper/2402.02932/) (2024)
- [AmharicIR+Instr: A Two-Dataset Resource for Neural Retrieval and Instruction Tuning](https://mlsanity.com/paper/2602.09914/) (2026)
- [Localization Boosting for Growth Markets: Mitigating Cross-Locale Behavioral Bias in Learning-to-Rank](https://mlsanity.com/paper/2605.11272/) (2026)
- [Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languages](https://mlsanity.com/paper/2607.12612/) (2026)

## Abstract (arXiv)

In this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a number of diverse language pairs. We first treat these models as multilingual text encoders and benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR. In contrast to supervised language understanding, our results indicate that for unsupervised document-level CLIR -- a setup with no relevance judgments for IR-specific fine-tuning -- pretrained multilingual encoders on average fail to significantly outperform earlier models based on CLWEs. For sentence-level retrieval, we do obtain state-of-the-art performance: the peak scores, however, are met by multilingual encoders that have been further specialized, in a supervised fashion, for sentence understanding tasks, rather than using their vanilla 'off-the-shelf' variants. Following these results, we introduce localized relevance matching for document-level CLIR, where we independently score a query against document sections. In the second part, we evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments. Our results show that supervised re-ranking rarely improves the performance of multilingual transformers as unsupervised base rankers. Finally, only with in-domain contrastive fine-tuning (i.e., same domain, only language transfer), we manage to improve the ranking quality. We uncover substantial empirical differences between cross-lingual retrieval results and results of (zero-shot) cross-lingual transfer for monolingual retrieval in target languages, which point to "monolingual overfitting" of retrieval models trained on monolingual data.