# Dynamic Dialogue Policy for Continual Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.05928 · HTML: https://mlsanity.com/paper/2204.05928/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 54 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [A continual learning survey: Defying forgetting in classification tasks](https://mlsanity.com/paper/1909.08383/) (2019)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Continual Learning in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2012.15504/) (2020)
- [Recent Advances and Challenges in Task-oriented Dialog System](https://mlsanity.com/paper/2003.07490/) (2020)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Continual Learning for Natural Language Generation in Task-oriented Dialog Systems](https://mlsanity.com/paper/2010.00910/) (2020)
- [Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning](https://mlsanity.com/paper/2003.05856/) (2020)
- [CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents](https://mlsanity.com/paper/2110.10067/) (2021)
- [Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog](https://mlsanity.com/paper/1908.10719/) (2019)
- [Optimizing for the Future in Non-Stationary MDPs](https://mlsanity.com/paper/2005.08158/) (2020)
- [Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version](https://mlsanity.com/paper/1904.10090/) (2019)
- [BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1711.05715/) (2017)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking](https://mlsanity.com/paper/2107.08173/) (2021)

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?](https://mlsanity.com/paper/2306.01386/) (2023)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via Dialogue](https://mlsanity.com/paper/2303.11108/) (2023)
- [A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions](https://mlsanity.com/paper/2504.16939/) (2025)
- [Infusing Emotions into Task-oriented Dialogue Systems: Understanding, Management, and Generation](https://mlsanity.com/paper/2408.02417/) (2024)
- [All Learning Has an Emotional Basis, So Does Task-Oriented Dialogue](https://mlsanity.com/paper/2507.01594/) (2025)
- [Intent-driven In-context Learning for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2412.03270/) (2024)
- [JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning](https://mlsanity.com/paper/2309.00230/) (2023)
- [Dialog Action-Aware Transformer for Dialog Policy Learning](https://mlsanity.com/paper/2309.02240/) (2023)
- [Regime-Adaptive Continual Learning for Portfolio Management](https://mlsanity.com/paper/2606.00143/) (2026)

## Abstract (arXiv)

Continual learning is one of the key components of human learning and a necessary requirement of artificial intelligence. As dialogue can potentially span infinitely many topics and tasks, a task-oriented dialogue system must have the capability to continually learn, dynamically adapting to new challenges while preserving the knowledge it already acquired. Despite the importance, continual reinforcement learning of the dialogue policy has remained largely unaddressed. The lack of a framework with training protocols, baseline models and suitable metrics, has so far hindered research in this direction. In this work we fill precisely this gap, enabling research in dialogue policy optimisation to go from static to dynamic learning. We provide a continual learning algorithm, baseline architectures and metrics for assessing continual learning models. Moreover, we propose the dynamic dialogue policy transformer (DDPT), a novel dynamic architecture that can integrate new knowledge seamlessly, is capable of handling large state spaces and obtains significant zero-shot performance when being exposed to unseen domains, without any growth in network parameter size.