# PaLM: A Hybrid Parser and Language Model

arXiv: https://arxiv.org/abs/1909.02134 · HTML: https://mlsanity.com/paper/1909.02134/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 13 of 13)

- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT](https://mlsanity.com/paper/2304.04920/) (2023)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [Pushdown Layers: Encoding Recursive Structure in Transformer Language Models](https://mlsanity.com/paper/2310.19089/) (2023)
- [Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling](https://mlsanity.com/paper/2112.07874/) (2021)
- [Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models](https://mlsanity.com/paper/2407.17406/) (2024)
- [Learning Representations on Logs for AIOps](https://mlsanity.com/paper/2308.11526/) (2023)
- [Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision](https://mlsanity.com/paper/2402.12691/) (2024)
- [Co-training an Unsupervised Constituency Parser with Weak Supervision](https://mlsanity.com/paper/2110.02283/) (2021)
- [Dependency-based Mixture Language Models](https://mlsanity.com/paper/2203.10256/) (2022)
- [Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic](https://mlsanity.com/paper/2510.17001/) (2025)
- [A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Models](https://mlsanity.com/paper/2602.09992/) (2026)
- [GiLT: Augmenting Transformer Language Models with Dependency Graphs](https://mlsanity.com/paper/2605.15562/) (2026)

## Abstract (arXiv)

We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.