# TegFormer: Topic-to-Essay Generation with Good Topic Coverage and High Text Coherence

arXiv: https://arxiv.org/abs/2212.13456 · HTML: https://mlsanity.com/paper/2212.13456/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 27 Dec 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?](https://mlsanity.com/paper/2106.01561/) (2021)
- [Exploring Controllable Text Generation Techniques](https://mlsanity.com/paper/2005.01822/) (2020)
- [Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2201.12438/) (2022)
- [CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model](https://mlsanity.com/paper/2003.01355/) (2020)
- [A comprehensive, application-oriented study of catastrophic forgetting in DNNs](https://mlsanity.com/paper/1905.08101/) (2019)
- [Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders](https://mlsanity.com/paper/1911.03882/) (2019)
- [Can Automatic Post-Editing Improve NMT?](https://mlsanity.com/paper/2009.14395/) (2020)
- [Overcoming Catastrophic Forgetting by Generative Regularization](https://mlsanity.com/paper/1912.01238/) (2019)
- [Controllable and Diverse Text Generation in E-commerce](https://mlsanity.com/paper/2102.11497/) (2021)
- [Generating Thematic Chinese Poetry using Conditional Variational Autoencoders with Hybrid Decoders](https://mlsanity.com/paper/1711.07632/) (2017)
- [Self-Supervised Learning for Contextualized Extractive Summarization](https://mlsanity.com/paper/1906.04466/) (2019)
- [A Sentiment-Controllable Topic-to-Essay Generator with Topic Knowledge Graph](https://mlsanity.com/paper/2010.05511/) (2020)
- [More Identifiable yet Equally Performant Transformers for Text Classification](https://mlsanity.com/paper/2106.01269/) (2021)
- [Topic-to-Essay Generation with Comprehensive Knowledge Enhancement](https://mlsanity.com/paper/2106.15142/) (2021)

## Abstract (arXiv)

Creating an essay based on a few given topics is a challenging NLP task. Although several effective methods for this problem, topic-to-essay generation, have appeared recently, there is still much room for improvement, especially in terms of the coverage of the given topics and the coherence of the generated text. In this paper, we propose a novel approach called TegFormer which utilizes the Transformer architecture where the encoder is enriched with domain-specific contexts while the decoder is enhanced by a large-scale pre-trained language model. Specifically, a Topic-Extension layer capturing the interaction between the given topics and their domain-specific contexts is plugged into the encoder. Since the given topics are usually concise and sparse, such an additional layer can bring more topic-related semantics in to facilitate the subsequent natural language generation. Moreover, an Embedding-Fusion module that combines the domain-specific word embeddings learnt from the given corpus and the general-purpose word embeddings provided by a GPT-2 model pre-trained on massive text data is integrated into the decoder. Since GPT-2 is at a much larger scale, it contains a lot more implicit linguistic knowledge which would help the decoder to produce more grammatical and readable text. Extensive experiments have shown that the pieces of text generated by TegFormer have better topic coverage and higher text coherence than those from SOTA topic-to-essay techniques, according to automatic and human evaluations. As revealed by ablation studies, both the Topic-Extension layer and the Embedding-Fusion module contribute substantially to TegFormer's performance advantage.