# Pretrained Language Models for Dialogue Generation with Multiple Input Sources

arXiv: https://arxiv.org/abs/2010.07576 · HTML: https://mlsanity.com/paper/2010.07576/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [Focused Attention Improves Document-Grounded Generation](https://mlsanity.com/paper/2104.12714/) (2021)
- [SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation](https://mlsanity.com/paper/2305.11130/) (2023)
- [Commonsense and Named Entity Aware Knowledge Grounded Dialogue Generation](https://mlsanity.com/paper/2205.13928/) (2022)
- [Long-term Control for Dialogue Generation: Methods and Evaluation](https://mlsanity.com/paper/2205.07352/) (2022)
- [A Survey of Personality, Persona, and Profile in Conversational Agents and Chatbots](https://mlsanity.com/paper/2401.00609/) (2023)
- [LLMR: Knowledge Distillation with a Large Language Model-Induced Reward](https://mlsanity.com/paper/2409.12500/) (2024)
- [Dynamically Retrieving Knowledge via Query Generation for Informative Dialogue Generation](https://mlsanity.com/paper/2208.00128/) (2022)
- [Multi-Step Dialogue Workflow Action Prediction](https://mlsanity.com/paper/2311.09593/) (2023)
- [Tgea: An error-annotated dataset and benchmark tasks for text generation from pretrained language models](https://mlsanity.com/paper/2503.04232/) (2025)
- [MSDF: A General Open-Domain Multi-Skill Dialog Framework](https://mlsanity.com/paper/2206.08626/) (2022)
- [UniRQR: A Unified Model for Retrieval Decision, Query, and Response Generation in Internet-Based Knowledge Dialogue Systems](https://mlsanity.com/paper/2401.06811/) (2024)

## Abstract (arXiv)

Large-scale pretrained language models have achieved outstanding performance on natural language understanding tasks. However, it is still under investigating how to apply them to dialogue generation tasks, especially those with responses conditioned on multiple sources. Previous work simply concatenates all input sources or averages information from different input sources. In this work, we study dialogue models with multiple input sources adapted from the pretrained language model GPT2. We explore various methods to fuse multiple separate attention information corresponding to different sources. Our experimental results show that proper fusion methods deliver higher relevance with dialogue history than simple fusion baselines.