# Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

arXiv: https://arxiv.org/abs/1909.06775 · HTML: https://mlsanity.com/paper/1909.06775/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2019.

## Built on by (most cited first, 30 of 46)

- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Consistency Regularization for Cross-Lingual Fine-Tuning](https://mlsanity.com/paper/2106.08226/) (2021)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers](https://mlsanity.com/paper/2306.02790/) (2023)
- [The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures](https://mlsanity.com/paper/2104.10640/) (2021)
- [Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks](https://mlsanity.com/paper/2108.08375/) (2021)
- [GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2204.08325/) (2022)
- [Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review](https://mlsanity.com/paper/2105.01044/) (2021)
- [X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering](https://mlsanity.com/paper/2104.09696/) (2021)
- [Analyzing Gender Representation in Multilingual Models](https://mlsanity.com/paper/2204.09168/) (2022)
- [Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining](https://mlsanity.com/paper/2105.10419/) (2021)
- [Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages](https://mlsanity.com/paper/2303.17683/) (2023)
- [El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsing](https://mlsanity.com/paper/2101.10524/) (2021)
- [Combining Static and Contextualised Multilingual Embeddings](https://mlsanity.com/paper/2203.09326/) (2022)
- [Transfer Language Selection for Zero-Shot Cross-Lingual Abusive Language Detection](https://mlsanity.com/paper/2206.00962/) (2022)
- [Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging](https://mlsanity.com/paper/2305.16834/) (2023)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces](https://mlsanity.com/paper/2210.05098/) (2022)
- [Polyglot Prompt: Multilingual Multitask PrompTraining](https://mlsanity.com/paper/2204.14264/) (2022)
- [ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training](https://mlsanity.com/paper/2211.08547/) (2022)
- [Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?](https://mlsanity.com/paper/2403.16777/) (2024)
- [Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation](https://mlsanity.com/paper/2103.02212/) (2021)
- [Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2110.08538/) (2021)
- [The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer](https://mlsanity.com/paper/2204.06457/) (2022)
- [On the Relation between Syntactic Divergence and Zero-Shot Performance](https://mlsanity.com/paper/2110.04644/) (2021)
- [Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations](https://mlsanity.com/paper/2307.13776/) (2023)
- [Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering](https://mlsanity.com/paper/2310.13583/) (2023)

## Abstract (arXiv)

This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018). In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the previous state-of-the-art that uses static embeddings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results.