# Changing the Mind of Transformers for Topically-Controllable Language Generation

arXiv: https://arxiv.org/abs/2103.15335 · HTML: https://mlsanity.com/paper/2103.15335/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [Event Representations for Automated Story Generation with Deep Neural Nets](https://mlsanity.com/paper/1706.01331/) (2017)
- [Polite Dialogue Generation Without Parallel Data](https://mlsanity.com/paper/1805.03162/) (2018)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [Controllable Neural Story Plot Generation via Reward Shaping](https://mlsanity.com/paper/1809.10736/) (2018)
- [A Controllable Model of Grounded Response Generation](https://mlsanity.com/paper/2005.00613/) (2020)
- [Plan, Write, and Revise: an Interactive System for Open-Domain Story Generation](https://mlsanity.com/paper/1904.02357/) (2019)
- [A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generation](https://mlsanity.com/paper/1808.06945/) (2018)
- [Story Realization: Expanding Plot Events into Sentences](https://mlsanity.com/paper/1909.03480/) (2019)
- [Progressive Generation of Long Text with Pretrained Language Models](https://mlsanity.com/paper/2006.15720/) (2020)
- [Generating Diverse Story Continuations with Controllable Semantics](https://mlsanity.com/paper/1909.13434/) (2019)
- [Learning to Predict Explainable Plots for Neural Story Generation](https://mlsanity.com/paper/1912.02395/) (2019)

## Built on by (most cited first, 7 of 7)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [A Plug-and-Play Method for Controlled Text Generation](https://mlsanity.com/paper/2109.09707/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generation](https://mlsanity.com/paper/2310.10706/) (2023)
- [Suggestion Lists vs. Continuous Generation: Interaction Design for Writing with Generative Models on Mobile Devices Affect Text Length, Wording and Perceived Authorship](https://mlsanity.com/paper/2208.00870/) (2022)
- [Uniform Complexity for Text Generation](https://mlsanity.com/paper/2204.05185/) (2022)

## Abstract (arXiv)

Large Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this limitation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the generation. Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and (2) a text generation model whose output adheres to the chosen topics. The training of both components is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework often generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers.