# A Bilingual Generative Transformer for Semantic Sentence Embedding

arXiv: https://arxiv.org/abs/1911.03895 · HTML: https://mlsanity.com/paper/1911.03895/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 16 of 16)

- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models](https://mlsanity.com/paper/2104.05115/) (2021)
- [MCSE: Multimodal Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2204.10931/) (2022)
- [Paraphrastic Representations at Scale](https://mlsanity.com/paper/2104.15114/) (2021)
- [A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond](https://mlsanity.com/paper/2305.12641/) (2023)
- [Language Models are Universal Embedders](https://mlsanity.com/paper/2310.08232/) (2023)
- [QA Is the New KR: Question-Answer Pairs as Knowledge Bases](https://mlsanity.com/paper/2207.00630/) (2022)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation](https://mlsanity.com/paper/2210.09773/) (2022)
- [Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval](https://mlsanity.com/paper/2212.10726/) (2022)
- [Joint Intent Detection And Slot Filling Based on Continual Learning Model](https://mlsanity.com/paper/2102.10905/) (2021)
- [The Daunting Dilemma with Sentence Encoders: Success on Standard Benchmarks, Failure in Capturing Basic Semantic Properties](https://mlsanity.com/paper/2309.03747/) (2023)
- [Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings](https://mlsanity.com/paper/2110.02030/) (2021)
- [Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?](https://mlsanity.com/paper/2511.20710/) (2025)
- [Distilling Monolingual and Crosslingual Word-in-Context Representations](https://mlsanity.com/paper/2409.08719/) (2024)

## Abstract (arXiv)

Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.