# Shallow Syntax in Deep Water

arXiv: https://arxiv.org/abs/1908.11047 · HTML: https://mlsanity.com/paper/1908.11047/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2019.

## Built on by (most cited first, 3 of 3)

- [Schr"odinger's Tree -- On Syntax and Neural Language Models](https://mlsanity.com/paper/2110.08887/) (2021)
- [Joint Universal Syntactic and Semantic Parsing](https://mlsanity.com/paper/2104.05696/) (2021)
- [Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2404.01992/) (2024)

## Abstract (arXiv)

Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain. We investigate the role of shallow syntax-aware representations for NLP tasks using two techniques. First, we enhance the ELMo architecture to allow pretraining on predicted shallow syntactic parses, instead of just raw text, so that contextual embeddings make use of shallow syntactic context. Our second method involves shallow syntactic features obtained automatically on downstream task data. Neither approach leads to a significant gain on any of the four downstream tasks we considered relative to ELMo-only baselines. Further analysis using black-box probes confirms that our shallow-syntax-aware contextual embeddings do not transfer to linguistic tasks any more easily than ELMo's embeddings. We take these findings as evidence that ELMo-style pretraining discovers representations which make additional awareness of shallow syntax redundant.