# Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework

arXiv: https://arxiv.org/abs/1910.04708 · HTML: https://mlsanity.com/paper/1910.04708/

## Key facts

- Cited by 26 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 10 Oct 2019.

## Built on by (most cited first, 26 of 26)

- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021)
- [Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings](https://mlsanity.com/paper/2210.12623/) (2022)
- [Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning](https://mlsanity.com/paper/2110.02600/) (2021)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction](https://mlsanity.com/paper/2106.03084/) (2021)
- [Subword Mapping and Anchoring across Languages](https://mlsanity.com/paper/2109.04556/) (2021)
- [Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast](https://mlsanity.com/paper/2109.00253/) (2021)
- [IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces](https://mlsanity.com/paper/2210.05098/) (2022)
- [Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation](https://mlsanity.com/paper/2104.04916/) (2021)
- [Majority Voting with Bidirectional Pre-translation For Bitext Retrieval](https://mlsanity.com/paper/2103.06369/) (2021)
- [Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts](https://mlsanity.com/paper/2204.04775/) (2022)
- [The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer](https://mlsanity.com/paper/2204.06457/) (2022)
- [Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?](https://mlsanity.com/paper/2411.04530/) (2024)
- [Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings](https://mlsanity.com/paper/2103.06459/) (2021)
- [Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study](https://mlsanity.com/paper/2106.04995/) (2021)
- [GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings](https://mlsanity.com/paper/2310.13068/) (2023)
- [Learning a Reversible Embedding Mapping using Bi-Directional Manifold Alignment](https://mlsanity.com/paper/2107.00124/) (2021)
- [Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks](https://mlsanity.com/paper/2102.00290/) (2021)
- [Isomorphic Cross-lingual Embeddings for Low-Resource Languages](https://mlsanity.com/paper/2203.14632/) (2022)
- [FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models](https://mlsanity.com/paper/2206.06586/) (2022)
- [Domain Mismatch Doesn't Always Prevent Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2211.16671/) (2022)
- [Multilingual Word Embeddings for Low-Resource Languages using Anchors and a Chain of Related Languages](https://mlsanity.com/paper/2311.12489/) (2023)
- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)
- [Cross-Domain Bilingual Lexicon Induction via Pretrained Language Models](https://mlsanity.com/paper/2505.23146/) (2025)
- [Semantic Alignment across Ancient Egyptian Language Stages via Normalization-Aware Multitask Learning](https://mlsanity.com/paper/2603.24258/) (2026)

## Abstract (arXiv)

Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent. Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches. Extensive experiments demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that this framework can generalize to contextualized representations such as Multilingual BERT, and produces state-of-the-art results on the CoNLL cross-lingual NER benchmark.