mlsanityData to 7 Oct 2026

Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages

arXiv 2610.02040 ↗First posted 1 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the exploration of this theory. In this paper, we extend existing analyses of the relationship between LMs' learning biases and typological commonality on both data and model sides, focusing on: (i) cross-serial dependencies, the upper limit of attested syntactic complexity, and (ii) stack-based LMs (SLMs), potentially facilitating learning of hierarchical patterns. We first evaluate generalization of SLMs on cross-serial dependencies across diverse artificial languages and confirm that they struggle with such constructions. However, SLMs with limited working memory generalize better suggesting a possible basis for such inductive bias and thus the typological commonality of some word order configurations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 49 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

18 of 49 references.

References · 18

  1. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  3. 2018Deep Learning: A Critical Appraisalcited by 214
  4. 2022Neural Networks and the Chomsky Hierarchycited by 214
  5. 2016Recurrent Neural Network Grammarscited by 109
  6. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  7. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  8. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  9. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  10. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  11. 2017The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languagescited by 19
  12. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
6 more
  1. 2024Kallini et al. (2024) do not compare impossible languages with constituency-based onescited by 15
  2. 2024Emergent Word Order Universals from Cognitively-Motivated Language Modelscited by 11
  3. 2025Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisitioncited by 10
  4. 2022The Surprising Computational Power of Nondeterministic Stack RNNscited by 6
  5. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  6. 2025Bearing Syntactic Fruit with Stack-Augmented Neural Networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
  2. 2026Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking
  3. 2026Language Models Generalize to Human-like Word Order Preferences
  4. 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?
  5. 2024Concept Space Alignment in Multilingual LLMs
  6. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  7. 2024Black Big Boxes: Tracing Adjective Order Preferences in Large Language Models
  8. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignment