# Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation

arXiv: https://arxiv.org/abs/2008.06788 · HTML: https://mlsanity.com/paper/2008.06788/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 30 of 34)

- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing](https://mlsanity.com/paper/2109.10540/) (2021)
- [Phylogeny-Inspired Adaptation of Multilingual Models to New Languages](https://mlsanity.com/paper/2205.09634/) (2022)
- [Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2110.08173/) (2021)
- [Romanization-based Large-scale Adaptation of Multilingual Language Models](https://mlsanity.com/paper/2304.08865/) (2023)
- [When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning](https://mlsanity.com/paper/2205.08124/) (2022)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP](https://mlsanity.com/paper/2111.09618/) (2021)
- [Multilingual Pretraining for Pixel Language Models](https://mlsanity.com/paper/2505.21265/) (2025)
- [Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking](https://mlsanity.com/paper/2109.13620/) (2021)
- [Local Structure Matters Most: Perturbation Study in NLU](https://mlsanity.com/paper/2107.13955/) (2021)
- [Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs](https://mlsanity.com/paper/2502.13195/) (2025)
- [Exploring the Landscape of Natural Language Processing Research](https://mlsanity.com/paper/2307.10652/) (2023)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [The futility of STILTs for the classification of lexical borrowings in Spanish](https://mlsanity.com/paper/2109.08607/) (2021)
- [Schr"odinger's Tree -- On Syntax and Neural Language Models](https://mlsanity.com/paper/2110.08887/) (2021)
- [Distilling Efficient Language-Specific Models for Cross-Lingual Transfer](https://mlsanity.com/paper/2306.01709/) (2023)
- [Have Attention Heads in BERT Learned Constituency Grammar?](https://mlsanity.com/paper/2102.07926/) (2021)
- [Joint Universal Syntactic and Semantic Parsing](https://mlsanity.com/paper/2104.05696/) (2021)
- [A Morphology-Based Investigation of Positional Encodings](https://mlsanity.com/paper/2404.04530/) (2024)
- [Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?](https://mlsanity.com/paper/2403.02009/) (2024)
- [COMBO: State-of-the-Art Morphosyntactic Analysis](https://mlsanity.com/paper/2109.05361/) (2021)
- [Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models](https://mlsanity.com/paper/2308.08774/) (2023)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [LySACoru~na at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing](https://mlsanity.com/paper/2204.12820/) (2022)

## Abstract (arXiv)

Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). The recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, however, questions this belief. In this work, we empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks. Relying on the established fine-tuning paradigm, we first couple a pretrained transformer with a biaffine parsing head, aiming to infuse explicit syntactic knowledge from Universal Dependencies treebanks into the transformer. We then fine-tune the model for LU tasks and measure the effect of the intermediate parsing training (IPT) on downstream LU task performance. Results from both monolingual English and zero-shot language transfer experiments (with intermediate target-language parsing) show that explicit formalized syntax, injected into transformers through IPT, has very limited and inconsistent effect on downstream LU performance. Our results, coupled with our analysis of transformers' representation spaces before and after intermediate parsing, make a significant step towards providing answers to an essential question: how (un)availing is supervised parsing for high-level semantic natural language understanding in the era of large neural models?