# Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

arXiv: https://arxiv.org/abs/1902.09492 · HTML: https://mlsanity.com/paper/1902.09492/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2019.

## Built on by (most cited first, 30 of 70)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages](https://mlsanity.com/paper/2402.12204/) (2024)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [Language Imbalance Driven Rewarding for Multilingual Self-improving](https://mlsanity.com/paper/2410.08964/) (2024)
- [Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention](https://mlsanity.com/paper/2410.12462/) (2024)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [Zero-Shot Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2104.07554/) (2021)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [MENLI: Robust Evaluation Metrics from Natural Language Inference](https://mlsanity.com/paper/2208.07316/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2202.10062/) (2022)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [Cluster & Tune: Boost Cold Start Performance in Text Classification](https://mlsanity.com/paper/2203.10581/) (2022)
- [Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction](https://mlsanity.com/paper/2106.03084/) (2021)
- [Learn and Unlearn: Addressing Misinformation in Multilingual LLMs](https://mlsanity.com/paper/2406.13748/) (2024)
- [Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing](https://mlsanity.com/paper/2104.04736/) (2021)
- [Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining](https://mlsanity.com/paper/2105.10419/) (2021)
- [Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy](https://mlsanity.com/paper/2106.13553/) (2021)
- [Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords](https://mlsanity.com/paper/2109.11491/) (2021)
- [El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsing](https://mlsanity.com/paper/2101.10524/) (2021)
- [Bootstrapping Multilingual AMR with Contextual Word Alignments](https://mlsanity.com/paper/2102.02189/) (2021)
- [Computational historical linguistics and language diversity in South Asia](https://mlsanity.com/paper/2203.12524/) (2022)
- [Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models](https://mlsanity.com/paper/2506.13044/) (2025)

## Abstract (arXiv)

We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.