# Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning

arXiv: https://arxiv.org/abs/2207.12185 · HTML: https://mlsanity.com/paper/2207.12185/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Recent Advances and Challenges in Task-oriented Dialog System](https://mlsanity.com/paper/2003.07490/) (2020)
- [Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1804.06512/) (2018)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Overview of the Ninth Dialog System Technology Challenge: DSTC9](https://mlsanity.com/paper/2011.06486/) (2020)
- [Structured Fusion Networks for Dialog](https://mlsanity.com/paper/1907.10016/) (2019)
- [Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog](https://mlsanity.com/paper/1908.10719/) (2019)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning](https://mlsanity.com/paper/1606.02560/) (2016)
- [MOSS: End-to-End Dialog System Framework with Modular Supervision](https://mlsanity.com/paper/1909.05528/) (2019)
- [End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.10712/) (2017)
- [Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation](https://mlsanity.com/paper/2005.07362/) (2020)

## Built on by (most cited first, 5 of 5)

- [Sparse Rewards Can Self-Train Dialogue Agents](https://mlsanity.com/paper/2409.04617/) (2024)
- [InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2310.08885/) (2023)
- [Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning](https://mlsanity.com/paper/2209.07873/) (2022)
- [Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization](https://mlsanity.com/paper/2305.03262/) (2023)
- [Team Flow at DRC2022: Pipeline System for Travel Destination Recommendation Task in Spoken Dialogue](https://mlsanity.com/paper/2210.09518/) (2022)

## Abstract (arXiv)

Many studies have proposed methods for optimizing the dialogue performance of an entire pipeline task-oriented dialogue system by jointly training modules in the system using reinforcement learning. However, these methods are limited in that they can only be applied to modules implemented using trainable neural-based methods. To solve this problem, we propose a method for optimizing a pipeline system composed of modules implemented with arbitrary methods for dialogue performance. With our method, neural-based components called post-processing networks (PPNs) are installed inside such a system to post-process the output of each module. All PPNs are updated to improve the overall dialogue performance of the system by using reinforcement learning, not necessitating each module to be differentiable. Through dialogue simulation and human evaluation on the MultiWOZ dataset, we show that our method can improve the dialogue performance of pipeline systems consisting of various modules.