mlsanityData to 7 Oct 2026

A Morphology-Based Investigation of Positional Encodings

arXiv 2404.04530 ↗First posted 6 Apr 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Contemporary deep learning models effectively handle languages with diverse morphology despite not being directly integrated into them. Morphology and word order are closely linked, with the latter incorporated into transformer-based models through positional encodings. This prompts a fundamental inquiry: Is there a correlation between the morphological complexity of a language and the utilization of positional encoding in pre-trained language models? In pursuit of an answer, we present the first study addressing this question, encompassing 22 languages and 5 downstream tasks. Our findings reveal that the importance of positional encoding diminishes with increasing morphological complexity in languages. Our study motivates the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 45 of its 99 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

45 of 99 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
33 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2018Deep contextualized word representationscited by 2,285
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  13. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  14. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  15. 2017Convolutional Sequence to Sequence Learningcited by 550
  16. 2019CamemBERT: a Tasty French Language Modelcited by 368
  17. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  18. 2019Assessing BERT's Syntactic Abilitiescited by 191
  19. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
  20. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  21. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  22. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  23. 2021Position Information in Transformers: An Overviewcited by 103
  24. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  25. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  26. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  27. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  28. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  29. 2022KinyaBERT: a Morphology-aware Kinyarwanda Language Modelcited by 27
  30. 2020A Systematic Analysis of Morphological Content in BERT Models for Multiple Languagescited by 20
  31. 2018Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representationscited by 18
  32. 2022What do complexity measures measure? Correlating and validating corpus-based measures of morphological complexitycited by 8
  33. 2019Robust Chinese Word Segmentation with Contextualized Word Representationscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  2. 2025MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoningcited by 2
  3. 2025On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibilitycited by 1
  4. 2024CSSL: Contrastive Self-Supervised Learning for Dependency Parsing on Relatively Free Word Ordered and Morphologically Rich Low Resource Languagescited by 0