# Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization

arXiv: https://arxiv.org/abs/2105.12544 · HTML: https://mlsanity.com/paper/2105.12544/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2010.01672/) (2020)
- [Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization](https://mlsanity.com/paper/1805.05271/) (2018)
- [Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts](https://mlsanity.com/paper/1809.05715/) (2018)
- [Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversations](https://mlsanity.com/paper/1910.01335/) (2019)
- [Co-GAT: A Co-Interactive Graph Attention Network for Joint Dialog Act Recognition and Sentiment Classification](https://mlsanity.com/paper/2012.13260/) (2020)
- [Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization](https://mlsanity.com/paper/2012.03502/) (2020)
- [Incorporating Commonsense Knowledge into Abstractive Dialogue Summarization via Heterogeneous Graph Networks](https://mlsanity.com/paper/2010.10044/) (2020)
- [How Domain Terminology Affects Meeting Summarization Performance](https://mlsanity.com/paper/2011.00692/) (2020)
- [Ask2Transformers: Zero-Shot Domain labelling with Pre-trained Language Models](https://mlsanity.com/paper/2101.02661/) (2021)

## Built on by (most cited first, 30 of 46)

- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [LLMaAA: Making Large Language Models as Active Annotators](https://mlsanity.com/paper/2310.19596/) (2023)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [A Survey on Dialogue Summarization: Recent Advances and New Frontiers](https://mlsanity.com/paper/2107.03175/) (2021)
- [CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning](https://mlsanity.com/paper/2112.08713/) (2021)
- [Controllable Neural Dialogue Summarization with Personal Named Entity Planning](https://mlsanity.com/paper/2109.13070/) (2021)
- [What Matters In The Structured Pruning of Generative Language Models?](https://mlsanity.com/paper/2302.03773/) (2023)
- [ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization](https://mlsanity.com/paper/2202.05599/) (2022)
- [TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision](https://mlsanity.com/paper/2403.00165/) (2024)
- [Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2203.01552/) (2022)
- [Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control](https://mlsanity.com/paper/2207.02263/) (2022)
- [Counseling Summarization using Mental Health Knowledge Guided Utterance Filtering](https://mlsanity.com/paper/2206.03886/) (2022)
- [VCSUM: A Versatile Chinese Meeting Summarization Dataset](https://mlsanity.com/paper/2305.05280/) (2023)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [Towards LifeSpan Cognitive Systems](https://mlsanity.com/paper/2409.13265/) (2024)
- [Response Generation with Context-Aware Prompt Learning](https://mlsanity.com/paper/2111.02643/) (2021)
- [Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization](https://mlsanity.com/paper/2409.14907/) (2024)
- [SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation](https://mlsanity.com/paper/2305.08371/) (2023)
- [Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization](https://mlsanity.com/paper/2305.16548/) (2023)
- [Abstractive Meeting Summarization: A Survey](https://mlsanity.com/paper/2208.04163/) (2022)
- [Towards Personalized Review Summarization by Modeling Historical Reviews from Customer and Product Separately](https://mlsanity.com/paper/2301.11682/) (2023)
- [The Cross-lingual Conversation Summarization Challenge](https://mlsanity.com/paper/2205.00379/) (2022)
- [Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling](https://mlsanity.com/paper/2204.08152/) (2022)
- [Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation](https://mlsanity.com/paper/2307.04018/) (2023)
- [Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization](https://mlsanity.com/paper/2310.10285/) (2023)
- [Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue Summarization](https://mlsanity.com/paper/2401.15496/) (2024)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)

## Abstract (arXiv)

Current dialogue summarization systems usually encode the text with a number of general semantic features (e.g., keywords and topics) to gain more powerful dialogue modeling capabilities. However, these features are obtained via open-domain toolkits that are dialog-agnostic or heavily relied on human annotations. In this paper, we show how DialoGPT, a pre-trained model for conversational response generation, can be developed as an unsupervised dialogue annotator, which takes advantage of dialogue background knowledge encoded in DialoGPT. We apply DialoGPT to label three types of features on two dialogue summarization datasets, SAMSum and AMI, and employ pre-trained and non pre-trained models as our summarizes. Experimental results show that our proposed method can obtain remarkable improvements on both datasets and achieves new state-of-the-art performance on the SAMSum dataset.