On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility
Not on the map. Why?
Abstract (arXiv)
Language model architectures are predominantly first created for English and subsequently applied to other languages. It is an open question whether this architectural bias leads to degraded performance for languages that are structurally different from English. We examine one specific architectural choice: positional encodings, through the lens of the trade-off hypothesis: the supposed interplay between morphological complexity and word order flexibility. This hypothesis posits a trade-off between the two: a more morphologically complex language can have a more flexible word order, and vice-versa. Positional encodings are a direct target to investigate the implications of this hypothesis in relation to language modelling. We pretrain monolingual model variants with absolute, relative, and no positional encodings for seven typologically diverse languages and evaluate them on four downstream tasks. Contrary to previous findings, we do not observe a clear interaction between position encodings and morphological complexity or word order flexibility, as measured by various proxies. Our results show that the choice of tasks, languages, and metrics are essential for drawing stable conclusions
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 11 Nov 2025.
Builds on
32 of 57 references.
References · 32
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
- 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
20 more
- 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
- 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
- 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
- 2021Position Information in Transformers: An Overviewcited by 103
- 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
- 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
- 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
- 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
- 2019Positional Encoding to Control Output Sequence Lengthcited by 56
- 2020What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encodingcited by 52
- 2019What Kind of Language Is Hard to Language-Model?cited by 50
- 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
- 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
- 2023A Better Way to Do Masked Language Model Scoringcited by 27
- 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
- 2024A Principled Framework for Evaluating on Typologically Diverse Languagescited by 12
- 2024Multilingual Gradient Word-Order Typology from Universal Dependenciescited by 9
- 2021The Impact of Positional Encodings on Multilingual Compressioncited by 5
- 2024A Morphology-Based Investigation of Positional Encodingscited by 4
- 2024How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLPcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.