# Response Generation with Context-Aware Prompt Learning

arXiv: https://arxiv.org/abs/2111.02643 · HTML: https://mlsanity.com/paper/2111.02643/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization](https://mlsanity.com/paper/2105.12544/) (2021)
- [DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder](https://mlsanity.com/paper/1805.12352/) (2018)
- [NSML: A Machine Learning Platform That Enables You to Focus on Your Models](https://mlsanity.com/paper/1712.05902/) (2017)
- [Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation](https://mlsanity.com/paper/2109.06513/) (2021)

## Built on by (most cited first, 12 of 12)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Instance-aware Prompt Learning for Language Understanding and Generation](https://mlsanity.com/paper/2201.07126/) (2022)
- [LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2402.04335/) (2024)
- [Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models](https://mlsanity.com/paper/2403.01754/) (2024)
- [MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension](https://mlsanity.com/paper/2310.18167/) (2023)
- [Enhancing Court View Generation with Knowledge Injection and Guidance](https://mlsanity.com/paper/2403.04366/) (2024)
- [Discourse-Aware Soft Prompting for Text Generation](https://mlsanity.com/paper/2112.05717/) (2021)
- [Prompting for a conversation: How to control a dialog model?](https://mlsanity.com/paper/2209.11068/) (2022)
- [Conversational Process Modeling: Can Generative AI Empower Domain Experts in Creating and Redesigning Process Models?](https://mlsanity.com/paper/2304.11065/) (2023)
- [AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code](https://mlsanity.com/paper/2507.19549/) (2025)
- [Attribute Controlled Dialogue Prompting](https://mlsanity.com/paper/2307.05228/) (2023)
- [K-ESConv: Knowledge Injection for Emotional Support Dialogue Systems via Prompt Learning](https://mlsanity.com/paper/2312.10371/) (2023)

## Abstract (arXiv)

Pre-trained language models (PLM) have marked a huge leap in neural dialogue modeling. While PLMs are pre-trained on large-scale text corpora, they are usually fine-tuned on scarce dialogue data with specific domain knowledge and dialogue styles. However, tailoring the language models while fully utilizing prior knowledge in large pre-trained models remains a challenge. In this paper, we present a novel approach for pre-trained dialogue modeling that casts the dialogue generation problem as a prompt-learning task. Instead of fine-tuning on limited dialogue data, our approach, DialogPrompt, learns continuous prompt embeddings optimized for dialogue contexts, which appropriately elicit knowledge from the large pre-trained model. To encourage the model to better utilize the prompt embeddings, the prompt encoders are designed to be dynamically generated based on the dialogue context. Experiments on popular conversation datasets show that our approach significantly outperforms the fine-tuning baseline and the generic prompt-learning methods. Furthermore, human evaluations strongly support the superiority of DialogPrompt in regard to response generation quality.