# Soft Prompt Decoding for Multilingual Dense Retrieval arXiv: https://arxiv.org/abs/2305.09025 ยท HTML: https://mlsanity.com/paper/2305.09025/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 58 of its 81 references that are in the map. - First posted on arXiv 15 May 2023. ## Builds on (most cited first, 30 of 58) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020) - [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021) - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016) - [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020) - [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021) - [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019) - [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020) - [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021) - [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022) - [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020) - [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017) - [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019) - [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018) - [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019) - [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020) - [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019) - [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021) - [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019) - [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020) - [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021) - [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020) - [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020) - [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021) - [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021) - [Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2010.02666/) (2020) ## Built on by (most cited first, 10 of 10) - [Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding](https://mlsanity.com/paper/2404.14600/) (2024) - [Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models](https://mlsanity.com/paper/2307.08303/) (2023) - [Language Fairness in Multilingual Information Retrieval](https://mlsanity.com/paper/2405.00978/) (2024) - [Distillation for Multilingual Information Retrieval](https://mlsanity.com/paper/2405.00977/) (2024) - [Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods](https://mlsanity.com/paper/2509.06195/) (2025) - [Synergistic Approach for Simultaneous Optimization of Monolingual, Cross-lingual, and Multilingual Information Retrieval](https://mlsanity.com/paper/2408.10536/) (2024) - [LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal](https://mlsanity.com/paper/2601.04768/) (2026) - [SPEAR: Soft Prompt Enhanced Anomaly Recognition for Time Series Data](https://mlsanity.com/paper/2510.03962/) (2025) - [SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval](https://mlsanity.com/paper/2606.18801/) (2026) - [Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval](https://mlsanity.com/paper/2608.02189/) (2026) ## Abstract (arXiv) In this work, we explore a Multilingual Information Retrieval (MLIR) task, where the collection includes documents in multiple languages. We demonstrate that applying state-of-the-art approaches developed for cross-lingual information retrieval to MLIR tasks leads to sub-optimal performance. This is due to the heterogeneous and imbalanced nature of multilingual collections -- some languages are better represented in the collection and some benefit from large-scale training data. To address this issue, we present KD-SPD, a novel soft prompt decoding approach for MLIR that implicitly "translates" the representation of documents in different languages into the same embedding space. To address the challenges of data scarcity and imbalance, we introduce a knowledge distillation strategy. The teacher model is trained on rich English retrieval data, and by leveraging bi-text data, our distillation framework transfers its retrieval knowledge to the multilingual document encoder. Therefore, our approach does not require any multilingual retrieval training data. Extensive experiments on three MLIR datasets with a total of 15 languages demonstrate that KD-SPD significantly outperforms competitive baselines in all cases. We conduct extensive analyses to show that our method has less language bias and better zero-shot transfer ability towards new languages.