Local Structure Matters Most in Most Languages
Not on the map. Why?
Abstract (arXiv)
Many recent perturbation studies have found unintuitive results on what does and does not matter when performing Natural Language Understanding (NLU) tasks in English. Coding properties, such as the order of words, can often be removed through shuffling without impacting downstream performances. Such insight may be used to direct future research into English NLP models. As many improvements in multilingual settings consist of wholesale adaptation of English approaches, it is important to verify whether those studies replicate or not in multilingual settings. In this work, we replicate a study on the importance of local structure, and the relative unimportance of global structure, in a multilingual setting. We find that the phenomenon observed on the English language broadly translates to over 120 languages, with a few caveats.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 9 Nov 2022.
Builds on
30 of 37 references.
References · 30
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2019Cross-lingual Language Model Pretrainingcited by 1,088
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2018Neural Network Acceptability Judgmentscited by 952
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
- 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
- 2019CamemBERT: a Tasty French Language Modelcited by 368
- 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
- 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
18 more
- 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
- 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
- 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
- 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
- 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
- 2019BERTje: A Dutch BERT Modelcited by 109
- 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
- 2020UnNatural Language Inferencecited by 68
- 2021What Context Features Can Transformer Language Models Use?cited by 49
- 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
- 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
- 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
- 2020PTT5: Pretraining and validating the T5 model on Brazilian Portuguese datacited by 35
- 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
- 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
- 2021Local Structure Matters Most: Perturbation Study in NLUcited by 10
- 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
- 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.