# Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment

arXiv: https://arxiv.org/abs/2404.02490 · HTML: https://mlsanity.com/paper/2404.02490/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax](https://mlsanity.com/paper/1902.08564/) (2019)
- [Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment](https://mlsanity.com/paper/2106.06381/) (2021)
- [Effective Parallel Corpus Mining using Bilingual Sentence Embeddings](https://mlsanity.com/paper/1807.11906/) (2018)
- [PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings](https://mlsanity.com/paper/2201.12093/) (2022)
- [A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT](https://mlsanity.com/paper/2004.14516/) (2020)
- [English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings](https://mlsanity.com/paper/2211.06127/) (2022)
- [WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction](https://mlsanity.com/paper/2306.05644/) (2023)
- [Dual-Alignment Pre-training for Cross-lingual Sentence Embedding](https://mlsanity.com/paper/2305.09148/) (2023)
- [Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding](https://mlsanity.com/paper/2309.08929/) (2023)
- [VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning](https://mlsanity.com/paper/2304.08205/) (2023)

## Built on by (most cited first, 13 of 13)

- [Word Alignment as Preference for Machine Translation](https://mlsanity.com/paper/2405.09223/) (2024)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Robust and Minimally Invasive Watermarking for EaaS](https://mlsanity.com/paper/2410.17552/) (2024)
- [Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings](https://mlsanity.com/paper/2509.12892/) (2025)
- [NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learning](https://mlsanity.com/paper/2601.03790/) (2026)
- [MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors](https://mlsanity.com/paper/2607.00848/) (2026)
- [Space Decomposition for Sentence Embedding](https://mlsanity.com/paper/2406.03125/) (2024)
- [LexPro-1.0 Technical Report](https://mlsanity.com/paper/2503.06949/) (2025)
- [CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages](https://mlsanity.com/paper/2509.16914/) (2025)
- [Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance](https://mlsanity.com/paper/2604.10590/) (2026)
- [Scaling Unsupervised Word Alignment to Documents via Structural Constraints](https://mlsanity.com/paper/2608.21023/) (2026)
- [Unsupervised Bilingual Lexicon Induction for Low Resource Languages](https://mlsanity.com/paper/2412.16894/) (2024)
- [GRC: Unifying Reasoning-Driven Generation, Retrieval and Compression](https://mlsanity.com/paper/2605.09100/) (2026)

## Abstract (arXiv)

The field of cross-lingual sentence embeddings has recently experienced significant advancements, but research concerning low-resource languages has lagged due to the scarcity of parallel corpora. This paper shows that cross-lingual word representation in low-resource languages is notably under-aligned with that in high-resource languages in current models. To address this, we introduce a novel framework that explicitly aligns words between English and eight low-resource languages, utilizing off-the-shelf word alignment models. This framework incorporates three primary training objectives: aligned word prediction and word translation ranking, along with the widely used translation ranking. We evaluate our approach through experiments on the bitext retrieval task, which demonstrate substantial improvements on sentence embeddings in low-resource languages. In addition, the competitive performance of the proposed model across a broader range of tasks in high-resource languages underscores its practicality.