# Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System

arXiv: https://arxiv.org/abs/2207.11762 · HTML: https://mlsanity.com/paper/2207.11762/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 24 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Recent Advances and Challenges in Task-oriented Dialog System](https://mlsanity.com/paper/2003.07490/) (2020)
- [A User Simulator for Task-Completion Dialogues](https://mlsanity.com/paper/1612.05688/) (2016)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems](https://mlsanity.com/paper/1807.11125/) (2018)
- [A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning](https://mlsanity.com/paper/2009.08115/) (2020)
- [Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1801.06176/) (2018)
- [Budgeted Policy Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1906.00499/) (2019)
- [Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1811.07550/) (2018)
- [Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Average](https://mlsanity.com/paper/1302.7175/) (2013)

## Built on by (most cited first, 8 of 8)

- [Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning](https://mlsanity.com/paper/2508.04848/) (2025)
- [A Generic Method for Fine-grained Category Discovery in Natural Language Texts](https://mlsanity.com/paper/2406.13103/) (2024)
- [Using Causality for Enhanced Prediction of Web Traffic Time Series](https://mlsanity.com/paper/2502.00612/) (2025)
- [Fighting Against the Repetitive Training and Sample Dependency Problem in Few-shot Named Entity Recognition](https://mlsanity.com/paper/2406.05460/) (2024)
- [Controllable Stylistic Text Generation with Train-Time Attribute-Regularized Diffusion](https://mlsanity.com/paper/2510.06386/) (2025)
- [Structured Information for Improving Spatial Relationships in Text-to-Image Generation](https://mlsanity.com/paper/2509.15962/) (2025)
- [Understanding Network Behaviors through Natural Language Question-Answering](https://mlsanity.com/paper/2510.21894/) (2025)
- [Automated Prompt Generation for Creative and Counterfactual Text-to-image Synthesis](https://mlsanity.com/paper/2509.21375/) (2025)

## Abstract (arXiv)

A dialogue policy module is an essential part of task-completion dialogue systems. Recently, increasing interest has focused on reinforcement learning (RL)-based dialogue policy. Its favorable performance and wise action decisions rely on an accurate estimation of action values. The overestimation problem is a widely known issue of RL since its estimate of the maximum action value is larger than the ground truth, which results in an unstable learning process and suboptimal policy. This problem is detrimental to RL-based dialogue policy learning. To mitigate this problem, this paper proposes a dynamic partial average estimator (DPAV) of the ground truth maximum action value. DPAV calculates the partial average between the predicted maximum action value and minimum action value, where the weights are dynamically adaptive and problem-dependent. We incorporate DPAV into a deep Q-network as the dialogue policy and show that our method can achieve better or comparable results compared to top baselines on three dialogue datasets of different domains with a lower computational load. In addition, we also theoretically prove the convergence and derive the upper and lower bounds of the bias compared with those of other methods.