# Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization

arXiv: https://arxiv.org/abs/2204.04362 · HTML: https://mlsanity.com/paper/2204.04362/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 9 Apr 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020)
- [Domain Agnostic Learning with Disentangled Representations](https://mlsanity.com/paper/1904.12347/) (2019)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2010.01672/) (2020)
- [Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation](https://mlsanity.com/paper/2010.12836/) (2020)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs](https://mlsanity.com/paper/2104.08400/) (2021)
- [Controllable Neural Dialogue Summarization with Personal Named Entity Planning](https://mlsanity.com/paper/2109.13070/) (2021)
- [TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising](https://mlsanity.com/paper/2001.00725/) (2020)
- [CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization](https://mlsanity.com/paper/2108.13139/) (2021)
- [Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling](https://mlsanity.com/paper/2012.07311/) (2020)
- [Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2109.04994/) (2021)
- [Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretraining](https://mlsanity.com/paper/2109.04080/) (2021)
- [TODSum: Task-Oriented Dialogue Summarization with State Tracking](https://mlsanity.com/paper/2110.12680/) (2021)
- [Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders](https://mlsanity.com/paper/2012.07300/) (2020)
- [Exploring Domain Shift in Extractive Text Summarization](https://mlsanity.com/paper/1908.11664/) (2019)
- [A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization](https://mlsanity.com/paper/1707.07062/) (2017)
- [Abstractive Summarization for Low Resource Data using Domain Transfer and Data Synthesis](https://mlsanity.com/paper/2002.03407/) (2020)

## Built on by (most cited first, 14 of 14)

- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Visually-Aware Context Modeling for News Image Captioning](https://mlsanity.com/paper/2308.08325/) (2023)
- [Compositional Zero-Shot Domain Transfer with Text-to-Text Models](https://mlsanity.com/paper/2303.13386/) (2023)
- [Few-shot Query-Focused Summarization with Prefix-Merging](https://mlsanity.com/paper/2211.16164/) (2022)
- [KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2306.15430/) (2023)
- [Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization](https://mlsanity.com/paper/2502.17328/) (2025)
- [OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System](https://mlsanity.com/paper/2312.16864/) (2023)
- [MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning](https://mlsanity.com/paper/2412.08946/) (2024)
- [Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning](https://mlsanity.com/paper/2305.12077/) (2023)
- [Unified Knowledge Prompt Pre-training for Customer Service Dialogues](https://mlsanity.com/paper/2208.14652/) (2022)
- [PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization](https://mlsanity.com/paper/2305.06647/) (2023)
- [Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning](https://mlsanity.com/paper/2408.17070/) (2024)
- [Domain Aligned Prefix Averaging for Domain Generalization in Abstractive Summarization](https://mlsanity.com/paper/2305.16820/) (2023)

## Abstract (arXiv)

The most advanced abstractive dialogue summarizers lack generalization ability on new domains and the existing researches for domain adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and generalizable Domain-Oriented Prefix-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalization. We conduct zero-shot experiments and build domain adaptation benchmarks on two multi-domain dialogue summarization datasets, TODSum and QMSum. Adequate experiments and qualitative analysis prove the effectiveness of our methods.