# Think Before You Speak: Explicitly Generating Implicit Commonsense Knowledge for Response Generation

arXiv: https://arxiv.org/abs/2110.08501 · HTML: https://mlsanity.com/paper/2110.08501/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 65 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [A Knowledge-Grounded Neural Conversation Model](https://mlsanity.com/paper/1702.01932/) (2017)
- [MuTual: A Dataset for Multi-Turn Dialogue Reasoning](https://mlsanity.com/paper/2004.04494/) (2020)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://mlsanity.com/paper/2010.08824/) (2020)
- [A Neural Network Approach to Context-Sensitive Generation of Conversational Responses](https://mlsanity.com/paper/1506.06714/) (2015)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)

## Built on by (most cited first, 26 of 26)

- [Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://mlsanity.com/paper/2305.15852/) (2023)
- [Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion](https://mlsanity.com/paper/2203.13224/) (2022)
- [Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents](https://mlsanity.com/paper/2310.09343/) (2023)
- [Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00930/) (2022)
- [Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinement](https://mlsanity.com/paper/2401.14215/) (2024)
- [Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior](https://mlsanity.com/paper/2503.17882/) (2025)
- [ComFact: A Benchmark for Linking Contextual Commonsense Knowledge](https://mlsanity.com/paper/2210.12678/) (2022)
- [Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue](https://mlsanity.com/paper/2311.07445/) (2023)
- [Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems](https://mlsanity.com/paper/2212.09180/) (2022)
- [Knowledge Enhanced Model for Live Video Comment Generation](https://mlsanity.com/paper/2304.14657/) (2023)
- [NewsDialogues: Towards Proactive News Grounded Conversation](https://mlsanity.com/paper/2308.06501/) (2023)
- [Norm Participation Grounds Language](https://mlsanity.com/paper/2206.02885/) (2022)
- [ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI](https://mlsanity.com/paper/2401.15471/) (2024)
- [Language Model Sentence Completion with a Parser-Driven Rhetorical Control Method](https://mlsanity.com/paper/2402.06125/) (2024)
- [Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model](https://mlsanity.com/paper/2411.04496/) (2024)
- [DiffuCOMET: Contextual Commonsense Knowledge Diffusion](https://mlsanity.com/paper/2402.17011/) (2024)
- [Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders](https://mlsanity.com/paper/2403.04314/) (2024)
- [What Really is Commonsense Knowledge?](https://mlsanity.com/paper/2411.03964/) (2024)
- [SkillQG: Learning to Generate Question for Reading Comprehension Assessment](https://mlsanity.com/paper/2305.04737/) (2023)
- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2402.02077/) (2024)
- [Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?](https://mlsanity.com/paper/2402.14355/) (2024)
- [Leveraging Explicit Reasoning for Inference Integration in Commonsense-Augmented Dialogue Models](https://mlsanity.com/paper/2406.09138/) (2024)
- [Towards Language-driven Scientific AI](https://mlsanity.com/paper/2210.15327/) (2022)
- [FairBelief -- Assessing Harmful Beliefs in Language Models](https://mlsanity.com/paper/2402.17389/) (2024)
- [FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition](https://mlsanity.com/paper/2403.00126/) (2024)

## Abstract (arXiv)

Implicit knowledge, such as common sense, is key to fluid human conversations. Current neural response generation (RG) models are trained to generate responses directly, omitting unstated implicit knowledge. In this paper, we present Think-Before-Speaking (TBS), a generative approach to first externalize implicit commonsense knowledge (think) and use this knowledge to generate responses (speak). We expect that externalizing implicit knowledge allows more efficient learning, produces more informative responses, and enables more explainable models. We analyze different choices to collect knowledge-aligned dialogues, represent implicit knowledge, and transition between knowledge and dialogues. Empirical results show TBS models outperform end-to-end and knowledge-augmented RG baselines on most automatic metrics and generate more informative, specific, and commonsense-following responses, as evaluated by human annotators. TBS also generates knowledge that makes sense and is relevant to the dialogue around 85% of the time.