# Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax

arXiv: https://arxiv.org/abs/1902.08564 · HTML: https://mlsanity.com/paper/1902.08564/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2019.

## Built on by (most cited first, 30 of 58)

- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages](https://mlsanity.com/paper/2205.12654/) (2022)
- [On the Evaluation of Vision-and-Language Navigation Instructions](https://mlsanity.com/paper/2101.10504/) (2021)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [Claim Matching Beyond English to Scale Global Fact-Checking](https://mlsanity.com/paper/2106.00853/) (2021)
- [CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs](https://mlsanity.com/paper/2112.08804/) (2021)
- [Few-shot Intent Classification and Slot Filling with Retrieved Examples](https://mlsanity.com/paper/2104.05763/) (2021)
- [Multilingual and Cross-Lingual Intent Detection from Spoken Data](https://mlsanity.com/paper/2104.08524/) (2021)
- [SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations](https://mlsanity.com/paper/2211.04508/) (2022)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [On Cross-Lingual Retrieval with Multilingual Text Encoders](https://mlsanity.com/paper/2112.11031/) (2021)
- [The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design](https://mlsanity.com/paper/2408.12503/) (2024)
- [Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval](https://mlsanity.com/paper/2101.08370/) (2021)
- [USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2202.10062/) (2022)
- [Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment](https://mlsanity.com/paper/2404.02490/) (2024)
- [LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation](https://mlsanity.com/paper/2302.08387/) (2023)
- [MINERS: Multilingual Language Models as Semantic Retrievers](https://mlsanity.com/paper/2406.07424/) (2024)
- [Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmark](https://mlsanity.com/paper/2306.07902/) (2023)
- [Dual-Alignment Pre-training for Cross-lingual Sentence Embedding](https://mlsanity.com/paper/2305.09148/) (2023)
- [SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives](https://mlsanity.com/paper/2306.02516/) (2023)
- [Towards Better Instruction Following Retrieval Models](https://mlsanity.com/paper/2505.21439/) (2025)
- [Assessing Reference-Free Peer Evaluation for Machine Translation](https://mlsanity.com/paper/2104.05146/) (2021)
- [NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2204.13021/) (2022)

## Abstract (arXiv)

In this paper, we present an approach to learn multilingual sentence embeddings using a bi-directional dual-encoder with additive margin softmax. The embeddings are able to achieve state-of-the-art results on the United Nations (UN) parallel corpus retrieval task. In all the languages tested, the system achieves P@1 of 86% or higher. We use pairs retrieved by our approach to train NMT models that achieve similar performance to models trained on gold pairs. We explore simple document-level embeddings constructed by averaging our sentence embeddings. On the UN document-level retrieval task, document embeddings achieve around 97% on P@1 for all experimented language pairs. Lastly, we evaluate the proposed model on the BUCC mining task. The learned embeddings with raw cosine similarity scores achieve competitive results compared to current state-of-the-art models, and with a second-stage scorer we achieve a new state-of-the-art level on this task.