# Transformer-Based Neural Text Generation with Syntactic Guidance

arXiv: https://arxiv.org/abs/2010.01737 · HTML: https://mlsanity.com/paper/2010.01737/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generators](https://mlsanity.com/paper/2205.12590/) (2022)
- [Continuous Decomposition of Granularity for Neural Paraphrase Generation](https://mlsanity.com/paper/2209.01765/) (2022)
- [Generative Adversarial Imitation Learning for Empathy-based AI](https://mlsanity.com/paper/2105.13328/) (2021)
- [Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?](https://mlsanity.com/paper/2205.06149/) (2022)

## Abstract (arXiv)

We study the problem of using (partial) constituency parse trees as syntactic guidance for controlled text generation. Existing approaches to this problem use recurrent structures, which not only suffer from the long-term dependency problem but also falls short in modeling the tree structure of the syntactic guidance. We propose to leverage the parallelism of Transformer to better incorporate parse trees. Our method first expands a partial template constituency parse tree to a full-fledged parse tree tailored for the input source text, and then uses the expanded tree to guide text generation. The effectiveness of our model in this process hinges upon two new attention mechanisms: 1) a path attention mechanism that forces one node to attend to only other nodes located in its path in the syntax tree to better incorporate syntax guidance; 2) a multi-encoder attention mechanism that allows the decoder to dynamically attend to information from multiple encoders. Our experiments in the controlled paraphrasing task show that our method outperforms SOTA models both semantically and syntactically, improving the best baseline's BLEU score from 11.83 to 26.27.