# Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer

arXiv: https://arxiv.org/abs/2105.05975 · HTML: https://mlsanity.com/paper/2105.05975/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Strong Baselines for Neural Semi-supervised Learning under Domain Shift](https://mlsanity.com/paper/1804.09530/) (2018)
- [MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://mlsanity.com/paper/1909.04761/) (2019)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations](https://mlsanity.com/paper/1910.05479/) (2019)
- [Survey on the Use of Typological Information in Natural Language Processing](https://mlsanity.com/paper/1610.03349/) (2016)

## Built on by (most cited first, 16 of 16)

- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021)
- [Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models](https://mlsanity.com/paper/2205.06130/) (2022)
- [Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Features](https://mlsanity.com/paper/2310.06458/) (2023)
- [Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space](https://mlsanity.com/paper/2305.02151/) (2023)
- [Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?](https://mlsanity.com/paper/2212.10879/) (2022)
- [Untangling the Influence of Typology, Data and Model Architecture on Ranking Transfer Languages for Cross-Lingual POS Tagging](https://mlsanity.com/paper/2503.19979/) (2025)
- [SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods](https://mlsanity.com/paper/2505.23714/) (2025)
- [Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes](https://mlsanity.com/paper/2211.05015/) (2022)
- [Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognition](https://mlsanity.com/paper/2309.05311/) (2023)
- [Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations](https://mlsanity.com/paper/2408.07599/) (2024)
- [Zero-shot Cross-lingual Transfer Learning with Multiple Source and Target Languages for Information Extraction: Language Selection and Adversarial Training](https://mlsanity.com/paper/2411.08785/) (2024)
- [Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models](https://mlsanity.com/paper/2502.11495/) (2025)
- [Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks](https://mlsanity.com/paper/2505.24834/) (2025)
- [Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models](https://mlsanity.com/paper/2604.08970/) (2026)
- [Language Distances are Practical for Equitable Cross-Lingual Transfer](https://mlsanity.com/paper/2609.32331/) (2026)

## Abstract (arXiv)

There is an increasing amount of evidence that in cases with little or no data in a target language, training on a different language can yield surprisingly good results. However, currently there are no established guidelines for choosing the training (source) language. In attempt to solve this issue we thoroughly analyze a state-of-the-art multilingual model and try to determine what impacts good transfer between languages. As opposed to the majority of multilingual NLP literature, we don't only train on English, but on a group of almost 30 languages. We show that looking at particular syntactic features is 2-4 times more helpful in predicting the performance than an aggregated syntactic similarity. We find out that the importance of syntactic features strongly differs depending on the downstream task - no single feature is a good performance predictor for all NLP tasks. As a result, one should not expect that for a target language L1 there is a single language L2 that is the best choice for any NLP task (for instance, for Bulgarian, the best source language is French on POS tagging, Russian on NER and Thai on NLI). We discuss the most important linguistic features affecting the transfer quality using statistical and machine learning methods.