# Does Typological Blinding Impede Cross-Lingual Sharing?

arXiv: https://arxiv.org/abs/2101.11888 · HTML: https://mlsanity.com/paper/2101.11888/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Zero-Shot Cross-Lingual Transfer with Meta Learning](https://mlsanity.com/paper/2003.02739/) (2020)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [The Limitations of Adversarial Training and the Blind-Spot Attack](https://mlsanity.com/paper/1901.04684/) (2019)
- [Stateful Detection of Black-Box Adversarial Attacks](https://mlsanity.com/paper/1907.05587/) (2019)
- [Inducing Language-Agnostic Multilingual Representations](https://mlsanity.com/paper/2008.09112/) (2020)
- [Learning Language Representations for Typology Prediction](https://mlsanity.com/paper/1707.09569/) (2017)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [Generalisation in Named Entity Recognition: A Quantitative Analysis](https://mlsanity.com/paper/1701.02877/) (2017)
- [What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties](https://mlsanity.com/paper/2009.12862/) (2020)
- [Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations](https://mlsanity.com/paper/2004.14923/) (2020)
- [Continuous multilinguality with language vectors](https://mlsanity.com/paper/1612.07486/) (2016)
- [What do Language Representations Really Represent?](https://mlsanity.com/paper/1901.02646/) (2019)
- [From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings](https://mlsanity.com/paper/1802.09375/) (2018)
- [SIGTYP 2020 Shared Task: Prediction of Typological Features](https://mlsanity.com/paper/2010.08246/) (2020)
- [A Probabilistic Generative Model of Linguistic Typology](https://mlsanity.com/paper/1903.10950/) (2019)
- [Tracking Typological Traits of Uralic Languages in Distributed Language Representations](https://mlsanity.com/paper/1711.05468/) (2017)
- [Survey on the Use of Typological Information in Natural Language Processing](https://mlsanity.com/paper/1610.03349/) (2016)
- [Parameter sharing between dependency parsers for related languages](https://mlsanity.com/paper/1808.09055/) (2018)
- [Uncovering Probabilistic Implications in Typological Knowledge Bases](https://mlsanity.com/paper/1906.07389/) (2019)

## Built on by (most cited first, 9 of 9)

- [Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?](https://mlsanity.com/paper/2212.10879/) (2022)
- [Language Embeddings Sometimes Contain Typological Generalizations](https://mlsanity.com/paper/2301.08115/) (2023)
- [Language Embeddings for Typology and Cross-lingual Transfer Learning](https://mlsanity.com/paper/2106.02082/) (2021)
- [Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization](https://mlsanity.com/paper/2310.12794/) (2023)
- [Word-order typology in Multilingual BERT: A case study in subordinate-clause detection](https://mlsanity.com/paper/2205.11987/) (2022)
- [Colexifications for Bootstrapping Cross-lingual Datasets: The Case of Phonology, Concreteness, and Affectiveness](https://mlsanity.com/paper/2306.02646/) (2023)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?](https://mlsanity.com/paper/2112.12356/) (2021)
- [Patterns of Persistence and Diffusibility across the World's Languages](https://mlsanity.com/paper/2401.01698/) (2024)

## Abstract (arXiv)

Bridging the performance gap between high- and low-resource languages has been the focus of much previous work. Typological features from databases such as the World Atlas of Language Structures (WALS) are a prime candidate for this, as such data exists even for very low-resource languages. However, previous work has only found minor benefits from using typological information. Our hypothesis is that a model trained in a cross-lingual setting will pick up on typological cues from the input data, thus overshadowing the utility of explicitly using such features. We verify this hypothesis by blinding a model to typological information, and investigate how cross-lingual sharing and performance is impacted. Our model is based on a cross-lingual architecture in which the latent weights governing the sharing between languages is learnt during training. We show that (i) preventing this model from exploiting typology severely reduces performance, while a control experiment reaffirms that (ii) encouraging sharing according to typology somewhat improves performance.