# WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue

arXiv: https://arxiv.org/abs/2108.01487 · HTML: https://mlsanity.com/paper/2108.01487/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 71 references that are in the map.
- First posted on arXiv 1 Aug 2021.

## Builds on (most cited first, 30 of 59)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems](https://mlsanity.com/paper/1701.03079/) (2017)
- [Continual Learning in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2012.15504/) (2020)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1704.03084/) (2017)
- [Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog](https://mlsanity.com/paper/1908.10719/) (2019)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances](https://mlsanity.com/paper/2012.01775/) (2020)
- [Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity](https://mlsanity.com/paper/2004.06577/) (2020)

## Built on by (most cited first, 2 of 2)

- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Towards Robust Online Dialogue Response Generation](https://mlsanity.com/paper/2203.03168/) (2022)

## Abstract (arXiv)

An intelligent dialogue system in a multi-turn setting should not only generate the responses which are of good quality, but it should also generate the responses which can lead to long-term success of the dialogue. Although, the current approaches improved the response quality, but they over-look the training signals present in the dialogue data. We can leverage these signals to generate the weakly supervised training data for learning dialog policy and reward estimator, and make the policy take actions (generates responses) which can foresee the future direction for a successful (rewarding) conversation. We simulate the dialogue between an agent and a user (modelled similar to an agent with supervised learning objective) to interact with each other. The agent uses dynamic blocking to generate ranked diverse responses and exploration-exploitation to select among the Top-K responses. Each simulated state-action pair is evaluated (works as a weak annotation) with three quality modules: Semantic Relevant, Semantic Coherence and Consistent Flow. Empirical studies with two benchmarks indicate that our model can significantly out-perform the response quality and lead to a successful conversation on both automatic evaluation and human judgement.