Better Benchmarking LLMs for Zero-Shot Dependency Parsing
Not on the map. Why?
Abstract (arXiv)
While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 28 Feb 2025.
Builds on
20 of 47 references.
References · 20
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
- 2021Meta-learning via Language Model In-context Tuningcited by 164
- 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
- 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
- 2022Prompting Language Models for Linguistic Structurecited by 39
- 2022Self-Prompting Large Language Models for Zero-Shot Open-Domain QAcited by 39
- 2019Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representationscited by 17
8 more
- 2021Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variablescited by 14
- 2021Minimum projective linearizations of trees in linear timecited by 9
- 2017Are crossing dependencies really scarce?cited by 7
- 2021Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformationcited by 5
- 2021Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performancecited by 5
- 2021The Linear Arrangement Library. A new tool for research on syntactic dependency structurescited by 5
- 2023ChatGPT is a Potential Zero-Shot Dependency Parsercited by 5
- 2023LLM-enhanced Self-training for Cross-domain Constituency Parsingcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.