# Word-order typology in Multilingual BERT: A case study in subordinate-clause detection

arXiv: https://arxiv.org/abs/2205.11987 · HTML: https://mlsanity.com/paper/2205.11987/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Is Multilingual BERT Fluent in Language Generation?](https://mlsanity.com/paper/1910.03806/) (2019)
- [A Survey on Text Simplification](https://mlsanity.com/paper/2008.08612/) (2020)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [Low-Resource Syntactic Transfer with Unsupervised Source Reordering](https://mlsanity.com/paper/1903.05683/) (2019)
- [Found in Translation: Reconstructing Phylogenetic Language Trees from Translations](https://mlsanity.com/paper/1704.07146/) (2017)
- [On the Relation between Syntactic Divergence and Zero-Shot Performance](https://mlsanity.com/paper/2110.04644/) (2021)
- [Discontinuous Constituency and BERT: A Case Study of Dutch](https://mlsanity.com/paper/2203.01063/) (2022)
- [Can pre-trained Transformers be used in detecting complex sensitive sentences? -- A Monsanto case study](https://mlsanity.com/paper/2203.06793/) (2022)

## Built on by (most cited first, 3 of 3)

- [Multilingual Gradient Word-Order Typology from Universal Dependencies](https://mlsanity.com/paper/2402.01513/) (2024)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering](https://mlsanity.com/paper/2310.13583/) (2023)

## Abstract (arXiv)

The capabilities and limitations of BERT and similar models are still unclear when it comes to learning syntactic abstractions, in particular across languages. In this paper, we use the task of subordinate-clause detection within and across languages to probe these properties. We show that this task is deceptively simple, with easy gains offset by a long tail of harder cases, and that BERT's zero-shot performance is dominated by word-order effects, mirroring the SVO/VSO/SOV typology.