LLMs' morphological analyses of complex FST-generated Finnish words
Not on the map. Why?
Abstract (arXiv)
Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 44 references that are in the map. First posted on arXiv 11 Jul 2024.
Builds on
19 of 44 references.
References · 19
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
- 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
- 2018Targeted Syntactic Evaluation of Language Modelscited by 239
- 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
- 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
- 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
- 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
7 more
- 2023FinGPT: Large Generative Models for a Small Languagecited by 49
- 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
- 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
- 2018Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debatecited by 27
- 2024Poro 34B and the Blessing of Multilingualitycited by 18
- 2019Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflectioncited by 13
- 2022UniMorph 4.0: Universal Morphologycited by 11
Built on by
Citing papers per year: 2025: 1.
1 paper.