# Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval

arXiv: https://arxiv.org/abs/2212.10726 · HTML: https://mlsanity.com/paper/2212.10726/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://mlsanity.com/paper/1909.05235/) (2019)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)
- [Spherical Latent Spaces for Stable Variational Autoencoders](https://mlsanity.com/paper/1808.10805/) (2018)
- [Towards Universal Paraphrastic Sentence Embeddings](https://mlsanity.com/paper/1511.08198/) (2015)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://mlsanity.com/paper/1704.04154/) (2017)
- [Effective Parallel Corpus Mining using Bilingual Sentence Embeddings](https://mlsanity.com/paper/1807.11906/) (2018)
- [Filtering and Mining Parallel Data in a Joint Multilingual Space](https://mlsanity.com/paper/1805.09822/) (2018)
- [Simple and Effective Paraphrastic Similarity from Parallel Translations](https://mlsanity.com/paper/1909.13872/) (2019)
- [Multilingual Models for Compositional Distributed Semantics](https://mlsanity.com/paper/1404.4641/) (2014)

## Built on by (most cited first, 5 of 5)

- [Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval](https://mlsanity.com/paper/2311.05800/) (2023)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2307.04096/) (2023)
- [Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment](https://mlsanity.com/paper/2407.14878/) (2024)
- [Explainable Disentangled Representation Learning for Generalizable Authorship Attribution in the Era of Generative AI](https://mlsanity.com/paper/2604.21300/) (2026)

## Abstract (arXiv)

Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes or careful engineering to work well. In this paper, we instead propose a generative model for learning multilingual text embeddings which can be used to retrieve or score sentence pairs. Our model operates on parallel data in N languages and, through an approximation we introduce, efficiently encourages source separation in this multilingual setting, separating semantic information that is shared between translations from stylistic or language-specific variation. We show careful large-scale comparisons between contrastive and generation-based approaches for learning multilingual text embeddings, a comparison that has not been done to the best of our knowledge despite the popularity of these approaches. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval -- the last of which we introduce in this paper. Overall, our Variational Multilingual Source-Separation Transformer (VMSST) model outperforms both a strong contrastive and generative baseline on these tasks.