# Leveraging Grammar Induction for Language Understanding and Generation

arXiv: https://arxiv.org/abs/2410.04878 · HTML: https://mlsanity.com/paper/2410.04878/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019)
- [Overcoming Language Priors in Visual Question Answering with Adversarial Regularization](https://mlsanity.com/paper/1810.03649/) (2018)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Neural Language Modeling by Jointly Learning Syntax and Lexicon](https://mlsanity.com/paper/1711.02013/) (2017)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa](https://mlsanity.com/paper/2104.04986/) (2021)
- [Straight to the Tree: Constituency Parsing with Neural Syntactic Distance](https://mlsanity.com/paper/1806.04168/) (2018)
- [Syntax-Enhanced Pre-trained Model](https://mlsanity.com/paper/2012.14116/) (2020)
- [Improving BERT with Syntax-aware Local Attention](https://mlsanity.com/paper/2012.15150/) (2020)
- [Enhancing Machine Translation with Dependency-Aware Self-Attention](https://mlsanity.com/paper/1909.03149/) (2019)
- [Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding](https://mlsanity.com/paper/1911.06156/) (2019)
- [Quantifying and Alleviating the Language Prior Problem in Visual Question Answering](https://mlsanity.com/paper/1905.04877/) (2019)
- [Improving Neural Language Models by Segmenting, Attending, and Predicting the Future](https://mlsanity.com/paper/1906.01702/) (2019)
- [Improving BERT Pretraining with Syntactic Supervision](https://mlsanity.com/paper/2104.10516/) (2021)
- [Syntax-guided Localized Self-attention by Constituency Syntactic Distance](https://mlsanity.com/paper/2210.11759/) (2022)
- [Inducing Constituency Trees through Neural Machine Translation](https://mlsanity.com/paper/1909.10056/) (2019)

## Built on by (most cited first, 1 of 1)

- [Know Yourself and Know Your Neighbour: A Syntactically Informed Self-Supervised Compositional Sentence Representation Learning Framework using a Recursive Hypernetwork](https://mlsanity.com/paper/2610.16547/) (2026)

## Abstract (arXiv)

Grammar induction has made significant progress in recent years. However, it is not clear how the application of induced grammar could enhance practical performance in downstream tasks. In this work, we introduce an unsupervised grammar induction method for language understanding and generation. We construct a grammar parser to induce constituency structures and dependency relations, which is simultaneously trained on downstream tasks without additional syntax annotations. The induced grammar features are subsequently incorporated into Transformer as a syntactic mask to guide self-attention. We evaluate and apply our method to multiple machine translation tasks and natural language understanding tasks. Our method demonstrates superior performance compared to the original Transformer and other models enhanced with external parsers. Experimental results indicate that our method is effective in both from-scratch and pre-trained scenarios. Additionally, our research highlights the contribution of explicitly modeling the grammatical structure of texts to neural network models.