# Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems

arXiv: https://arxiv.org/abs/2109.04084 · HTML: https://mlsanity.com/paper/2109.04084/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [Proactive Human-Machine Conversation with Explicit Conversation Goals](https://mlsanity.com/paper/1906.05572/) (2019)
- [Target-Guided Open-Domain Conversation](https://mlsanity.com/paper/1905.11553/) (2019)
- [Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs](https://mlsanity.com/paper/1911.02707/) (2019)
- [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018)
- [Insertion-based Decoding with automatically Inferred Generation Order](https://mlsanity.com/paper/1902.01370/) (2019)
- [Generate, Delete and Rewrite: A Three-Stage Framework for Improving Persona Consistency of Dialogue Generation](https://mlsanity.com/paper/2004.07672/) (2020)
- [Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation](https://mlsanity.com/paper/1607.00970/) (2016)
- [PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation](https://mlsanity.com/paper/2010.02301/) (2020)
- [ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation](https://mlsanity.com/paper/1907.05339/) (2019)
- [Dynamic Knowledge Routing Network For Target-Guided Open-Domain Conversation](https://mlsanity.com/paper/2002.01196/) (2020)

## Built on by (most cited first, 17 of 17)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation](https://mlsanity.com/paper/2208.08110/) (2022)
- [Emp-RFT: Empathetic Response Generation via Recognizing Feature Transitions between Utterances](https://mlsanity.com/paper/2205.03112/) (2022)
- [An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation](https://mlsanity.com/paper/2209.14627/) (2022)
- [SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation](https://mlsanity.com/paper/2305.11130/) (2023)
- [Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue](https://mlsanity.com/paper/2311.07445/) (2023)
- [C3KG: A Chinese Commonsense Conversation Knowledge Graph](https://mlsanity.com/paper/2204.02549/) (2022)
- [Empathetic Response Generation via Emotion Cause Transition Graph](https://mlsanity.com/paper/2302.11787/) (2023)
- [Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues](https://mlsanity.com/paper/2210.16838/) (2022)
- [State-of-the-art in Open-domain Conversational AI: A Survey](https://mlsanity.com/paper/2205.00965/) (2022)
- [T5 for Hate Speech, Augmented Data and Ensemble](https://mlsanity.com/paper/2210.05480/) (2022)
- [The Future of Cognitive Strategy-enhanced Persuasive Dialogue Agents: New Perspectives and Trends](https://mlsanity.com/paper/2402.04631/) (2024)
- [Partner Personas Generation for Diverse Dialogue Generation](https://mlsanity.com/paper/2111.13833/) (2021)
- [Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2402.02077/) (2024)
- [Personalized Topic Selection Model for Topic-Grounded Dialogue](https://mlsanity.com/paper/2406.01988/) (2024)
- [Toxic Subword Pruning for Dialogue Response Generation on Large Language Models](https://mlsanity.com/paper/2410.04155/) (2024)
- [Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation](https://mlsanity.com/paper/2502.04537/) (2025)

## Abstract (arXiv)

Human dialogue contains evolving concepts, and speakers naturally associate multiple concepts to compose a response. However, current dialogue models with the seq2seq framework lack the ability to effectively manage concept transitions and can hardly introduce multiple concepts to responses in a sequential decoding manner. To facilitate a controllable and coherent dialogue, in this work, we devise a concept-guided non-autoregressive model (CG-nAR) for open-domain dialogue generation. The proposed model comprises a multi-concept planning module that learns to identify multiple associated concepts from a concept graph and a customized Insertion Transformer that performs concept-guided non-autoregressive generation to complete a response. The experimental results on two public datasets show that CG-nAR can produce diverse and coherent responses, outperforming state-of-the-art baselines in both automatic and human evaluations with substantially faster inference speed.