# Better Benchmarking LLMs for Zero-Shot Dependency Parsing

arXiv: https://arxiv.org/abs/2502.20866 · HTML: https://mlsanity.com/paper/2502.20866/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://mlsanity.com/paper/2306.04136/) (2023)
- [Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking](https://mlsanity.com/paper/2310.13243/) (2023)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [Self-Prompting Large Language Models for Zero-Shot Open-Domain QA](https://mlsanity.com/paper/2212.08635/) (2022)
- [Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations](https://mlsanity.com/paper/1910.05479/) (2019)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [Minimum projective linearizations of trees in linear time](https://mlsanity.com/paper/2102.03277/) (2021)
- [Are crossing dependencies really scarce?](https://mlsanity.com/paper/1703.08324/) (2017)
- [Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation](https://mlsanity.com/paper/2103.02212/) (2021)
- [Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performance](https://mlsanity.com/paper/2106.00352/) (2021)
- [The Linear Arrangement Library. A new tool for research on syntactic dependency structures](https://mlsanity.com/paper/2112.02512/) (2021)
- [ChatGPT is a Potential Zero-Shot Dependency Parser](https://mlsanity.com/paper/2310.16654/) (2023)
- [LLM-enhanced Self-training for Cross-domain Constituency Parsing](https://mlsanity.com/paper/2311.02660/) (2023)

## Built on by (most cited first, 2 of 2)

- [Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models](https://mlsanity.com/paper/2603.16718/) (2026)
- [Synthetic Data Characterization via Training Dynamics](https://mlsanity.com/paper/2609.39447/) (2026)

## Abstract (arXiv)

While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs.