# Probing for Labeled Dependency Trees

arXiv: https://arxiv.org/abs/2203.12971 · HTML: https://mlsanity.com/paper/2203.12971/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [A Weighted Correlation Index for Rankings with Ties](https://mlsanity.com/paper/1404.3325/) (2014)
- [A Tale of a Probe and a Parser](https://mlsanity.com/paper/2005.01641/) (2020)
- [Do Neural Language Models Show Preferences for Syntactic Formalisms?](https://mlsanity.com/paper/2004.14096/) (2020)
- [Introducing Orthogonal Constraint in Structural Probes](https://mlsanity.com/paper/2012.15228/) (2020)
- [Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing -- A Tale of Two Parsers Revisited](https://mlsanity.com/paper/1908.07397/) (2019)

## Built on by (most cited first, 8 of 8)

- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [Evidence > Intuition: Transferability Estimation for Encoder Selection](https://mlsanity.com/paper/2210.11255/) (2022)
- [Probing Syntax in Large Language Models: Successes and Remaining Challenges](https://mlsanity.com/paper/2508.03211/) (2025)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Assessment of Pre-Trained Models Across Languages and Grammars](https://mlsanity.com/paper/2309.11165/) (2023)
- [Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks](https://mlsanity.com/paper/2601.18617/) (2026)
- [Polar probe linearly decodes semantic structures from LLMs](https://mlsanity.com/paper/2605.14125/) (2026)

## Abstract (arXiv)

Probing has become an important tool for analyzing representations in Natural Language Processing (NLP). For graphical NLP tasks such as dependency parsing, linear probes are currently limited to extracting undirected or unlabeled parse trees which do not capture the full task. This work introduces DepProbe, a linear probe which can extract labeled and directed dependency parse trees from embeddings while using fewer parameters and compute than prior methods. Leveraging its full task coverage and lightweight parametrization, we investigate its predictive power for selecting the best transfer language for training a full biaffine attention parser. Across 13 languages, our proposed method identifies the best source treebank 94% of the time, outperforming competitive baselines and prior work. Finally, we analyze the informativeness of task-specific subspaces in contextual embeddings as well as which benefits a full parser's non-linear parametrization provides.