# Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering

arXiv: https://arxiv.org/abs/2310.13583 · HTML: https://mlsanity.com/paper/2310.13583/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/1909.06775/) (2019)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [Don't paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing](https://mlsanity.com/paper/1908.09940/) (2019)
- [The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languages](https://mlsanity.com/paper/1710.03838/) (2017)
- [Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages](https://mlsanity.com/paper/1811.00383/) (2018)
- [Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing](https://mlsanity.com/paper/1909.01482/) (2019)
- [Fine-Grained Analysis of Cross-Linguistic Syntactic Divergences](https://mlsanity.com/paper/2005.03436/) (2020)
- [Cross-Lingual Syntactic Transfer with Limited Resources](https://mlsanity.com/paper/1610.06227/) (2016)
- [Low-Resource Syntactic Transfer with Unsupervised Source Reordering](https://mlsanity.com/paper/1903.05683/) (2019)
- [A Data Bootstrapping Recipe for Low Resource Multilingual Relation Classification](https://mlsanity.com/paper/2110.09570/) (2021)
- [Multilingual Neural Semantic Parsing for Low-Resourced Languages](https://mlsanity.com/paper/2106.03469/) (2021)

## Built on by (most cited first, 4 of 4)

- [Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective](https://mlsanity.com/paper/2410.07239/) (2024)
- [Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations](https://mlsanity.com/paper/2408.07599/) (2024)
- [Implicit Word Reordering with Knowledge Distillation for Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2502.17308/) (2025)
- [Zero-shot Dependency Parsing with Unsupervised Cross-Lingual Bootstrapping](https://mlsanity.com/paper/2609.37883/) (2026)

## Abstract (arXiv)

Despite the impressive growth of the abilities of multilingual language models, such as XLM-R and mT5, it has been shown that they still face difficulties when tackling typologically-distant languages, particularly in the low-resource setting. One obstacle for effective cross-lingual transfer is variability in word-order patterns. It can be potentially mitigated via source- or target-side word reordering, and numerous approaches to reordering have been proposed. However, they rely on language-specific rules, work on the level of POS tags, or only target the main clause, leaving subordinate clauses intact. To address these limitations, we present a new powerful reordering method, defined in terms of Universal Dependencies, that is able to learn fine-grained word-order patterns conditioned on the syntactic context from a small amount of annotated data and can be applied at all levels of the syntactic tree. We conduct experiments on a diverse set of tasks and show that our method consistently outperforms strong baselines over different language pairs and model architectures. This performance advantage holds true in both zero-shot and few-shot scenarios.