# Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation

arXiv: https://arxiv.org/abs/2209.04179 · HTML: https://mlsanity.com/paper/2209.04179/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 9 Sep 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [A Survey of Knowledge-Enhanced Text Generation](https://mlsanity.com/paper/2010.04389/) (2020)
- [Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2002.07510/) (2020)
- [Neural Question Generation from Text: A Preliminary Study](https://mlsanity.com/paper/1704.01792/) (2017)
- [Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering](https://mlsanity.com/paper/1909.06356/) (2019)
- [Story Ending Generation with Incremental Encoding and Commonsense Knowledge](https://mlsanity.com/paper/1808.10113/) (2018)
- [Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation](https://mlsanity.com/paper/1908.04942/) (2019)
- [Generating Natural Questions About an Image](https://mlsanity.com/paper/1603.06059/) (2016)
- [Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia](https://mlsanity.com/paper/1805.05942/) (2018)
- [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018)
- [ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation](https://mlsanity.com/paper/2001.11314/) (2020)
- [Semantic Graphs for Generating Deep Questions](https://mlsanity.com/paper/2004.12704/) (2020)
- [Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation](https://mlsanity.com/paper/1607.00970/) (2016)
- [Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus](https://mlsanity.com/paper/1603.06807/) (2016)
- [Improving Neural Question Generation using Answer Separation](https://mlsanity.com/paper/1809.02393/) (2018)
- [RefNet: A Reference-aware Network for Background Based Conversation](https://mlsanity.com/paper/1908.06449/) (2019)
- [Let's Ask Again: Refine Network for Automatic Question Generation](https://mlsanity.com/paper/1909.05355/) (2019)
- [Improving Question Generation with Sentence-level Semantic Matching and Answer Position Inferring](https://mlsanity.com/paper/1912.00879/) (2019)

## Built on by (most cited first, 1 of 1)

- [Ask To The Point: Open-Domain Entity-Centric Question Generation](https://mlsanity.com/paper/2310.14126/) (2023)

## Abstract (arXiv)

Today the pre-trained language models achieve great success for question generation (QG) task and significantly outperform traditional sequence-to-sequence approaches. However, the pre-trained models treat the input passage as a flat sequence and are thus not aware of the text structure of input passage. For QG task, we model text structure as answer position and syntactic dependency, and propose answer localness modeling and syntactic mask attention to address these limitations. Specially, we present localness modeling with a Gaussian bias to enable the model to focus on answer-surrounded context, and propose a mask attention mechanism to make the syntactic structure of input passage accessible in question generation process. Experiments on SQuAD dataset show that our proposed two modules improve performance over the strong pre-trained model ProphetNet, and combing them together achieves very competitive results with the state-of-the-art pre-trained model.