Assessment of Pre-Trained Models Across Languages and Grammars
Not on the map. Why?
Abstract (arXiv)
We present an approach for assessing how multilingual large language models (LLMs) learn syntax in terms of multi-formalism syntactic structures. We aim to recover constituent and dependency structures by casting parsing as sequence labeling. To do so, we select a few LLMs and study them on 13 diverse UD treebanks for dependency parsing and 10 treebanks for constituent parsing. Our results show that: (i) the framework is consistent across encodings, (ii) pre-trained word vectors do not favor constituency representations of syntax over dependencies, (iii) sub-word tokenization is needed to represent syntax, in contrast to character-based models, and (iv) occurrence of a language in the pretraining data is more important than the amount of task data when recovering syntax from the word vectors.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 20 Sep 2023.
Builds on
35 of 55 references.
References · 35
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
- 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
- 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
23 more
- 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
- 2018Constituency Parsing with a Self-Attentive Encodercited by 199
- 2019Multilingual is not enough: BERT for Finnishcited by 119
- 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
- 2016Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representationscited by 55
- 2019Viable Dependency Parsing as Sequence Labelingcited by 34
- 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?cited by 30
- 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
- 2020Give your Text Representation Models some Love: the Case for Basquecited by 28
- 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
- 2018Constituent Parsing as Sequence Labelingcited by 26
- 2020Parsing as Pretrainingcited by 19
- 2019Tetra-Tagging: Word-Synchronous Parsing with Linear-Time Inferencecited by 17
- 2020Bracketing Encodings for 2-Planar Dependency Parsingcited by 15
- 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
- 2020A Unifying Theory of Transition-based and Sequence Labeling Parsingcited by 12
- 2022Probing for Labeled Dependency Treescited by 8
- 2021Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsingcited by 7
- 2022Finding Structural Knowledge in Multimodal-BERTcited by 7
- 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
- 2017Fast(er) Exact Decoding and Global Training for Transition-Based Dependency Parsing via a Minimal Feature Setcited by 5
- 2017Generic Axiomatization of Families of Noncrossing Graphs in Dependency Parsingcited by 4
- 2022Sort by Structure: Language Model Ranking as Dependency Probingcited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.