# DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances

arXiv: https://arxiv.org/abs/2012.01775 · HTML: https://mlsanity.com/paper/2012.01775/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 30 of 36)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings](https://mlsanity.com/paper/2305.02317/) (2023)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [Task Compass: Scaling Multi-task Pre-training with Task Prefix](https://mlsanity.com/paper/2210.06277/) (2022)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2103.03125/) (2021)
- [Emp-RFT: Empathetic Response Generation via Recognizing Feature Transitions between Utterances](https://mlsanity.com/paper/2205.03112/) (2022)
- [An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation](https://mlsanity.com/paper/2209.14627/) (2022)
- [Teacher Forcing Recovers Reward Functions for Text Generation](https://mlsanity.com/paper/2210.08708/) (2022)
- [Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue](https://mlsanity.com/paper/2311.07445/) (2023)
- [Towards Personalized Review Summarization by Modeling Historical Reviews from Customer and Product Separately](https://mlsanity.com/paper/2301.11682/) (2023)
- [An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets](https://mlsanity.com/paper/2201.06219/) (2022)
- [Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation](https://mlsanity.com/paper/2408.02976/) (2024)
- [Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling](https://mlsanity.com/paper/2204.08152/) (2022)
- [Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation](https://mlsanity.com/paper/2404.02589/) (2024)
- [Smoothing Dialogue States for Open Conversational Machine Reading](https://mlsanity.com/paper/2108.12599/) (2021)
- [Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure](https://mlsanity.com/paper/2210.12461/) (2022)
- [IMAD: IMage-Augmented multi-modal Dialogue](https://mlsanity.com/paper/2305.10512/) (2023)
- [KPT: Keyword-guided Pre-training for Grounded Dialog Generation](https://mlsanity.com/paper/2212.01739/) (2022)
- [UniRetriever: Multi-task Candidates Selection for Various Context-Adaptive Conversational Retrieval](https://mlsanity.com/paper/2402.16261/) (2024)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Unified Knowledge Prompt Pre-training for Customer Service Dialogues](https://mlsanity.com/paper/2208.14652/) (2022)
- [DialoGen: Generalized Long-Range Context Representation for Dialogue Systems](https://mlsanity.com/paper/2210.06282/) (2022)
- [Discovering Elementary Discourse Units in Textual Data Using Canonical Correlation Analysis](https://mlsanity.com/paper/2406.12997/) (2024)
- ["No, they did not": Dialogue response dynamics in pre-trained language models](https://mlsanity.com/paper/2210.02526/) (2022)
- [Evaluating and Improving Context Attention Distribution on Multi-Turn Response Generation using Self-Contained Distractions](https://mlsanity.com/paper/2211.04943/) (2022)
- [An Empirical Study of Multitask Learning to Improve Open Domain Dialogue Systems](https://mlsanity.com/paper/2304.08115/) (2023)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2110.04984/) (2021)
- [LAME: Layout Aware Metadata Extraction Approach for Research Articles](https://mlsanity.com/paper/2112.12353/) (2021)
- [DSTEA: Improving Dialogue State Tracking via Entity Adaptive Pre-training](https://mlsanity.com/paper/2207.03858/) (2022)

## Abstract (arXiv)

Recent advances in pre-trained language models have significantly improved neural response generation. However, existing methods usually view the dialogue context as a linear sequence of tokens and learn to generate the next word through token-level self-attention. Such token-level encoding hinders the exploration of discourse-level coherence among utterances. This paper presents DialogBERT, a novel conversational response generation model that enhances previous PLM-based dialogue models. DialogBERT employs a hierarchical Transformer architecture. To efficiently capture the discourse-level coherence among utterances, we propose two training objectives, including masked utterance regression and distributed utterance order ranking in analogy to the original BERT training. Experiments on three multi-turn conversation datasets show that our approach remarkably outperforms the baselines, such as BART and DialoGPT, in terms of quantitative evaluation. The human evaluation suggests that DialogBERT generates more coherent, informative, and human-like responses than the baselines with significant margins.