# Reward-Driven Interaction: Enhancing Proactive Dialogue Agents through User Satisfaction Prediction

arXiv: https://arxiv.org/abs/2505.18731 · HTML: https://mlsanity.com/paper/2505.18731/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 21 of 21)

- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [Joint Slot Filling and Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1812.09471/) (2018)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition](https://mlsanity.com/paper/2207.00193/) (2022)
- [Ask the Right Questions: Active Question Reformulation with Reinforcement Learning](https://mlsanity.com/paper/1705.07830/) (2017)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [MIMICS: A Large-Scale Data Collection for Search Clarification](https://mlsanity.com/paper/2006.10174/) (2020)
- [Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model](https://mlsanity.com/paper/1907.02884/) (2019)
- [AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence](https://mlsanity.com/paper/2502.13943/) (2025)
- [Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations](https://mlsanity.com/paper/2010.02495/) (2020)
- [Feedback-Based Self-Learning in Large-Scale Conversational AI Agents](https://mlsanity.com/paper/1911.02557/) (2019)
- [Non-Autoregressive Neural Text-to-Speech](https://mlsanity.com/paper/1905.08459/) (2019)
- [Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing](https://mlsanity.com/paper/2008.11087/) (2020)
- [RePreM: Representation Pre-training with Masked Model for Reinforcement Learning](https://mlsanity.com/paper/2303.01668/) (2023)
- [TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets](https://mlsanity.com/paper/2212.02125/) (2022)
- [Design Considerations For Hypothesis Rejection Modules In Spoken Language Understanding Systems](https://mlsanity.com/paper/2211.09711/) (2022)
- [A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS](https://mlsanity.com/paper/2212.03817/) (2022)

## Abstract (arXiv)

Reward-driven proactive dialogue agents require precise estimation of user satisfaction as an intrinsic reward signal to determine optimal interaction strategies. Specifically, this framework triggers clarification questions when detecting potential user dissatisfaction during interactions in the industrial dialogue system. Traditional works typically rely on training a neural network model based on weak labels which are generated by a simple model trained on user actions after current turn. However, existing methods suffer from two critical limitations in real-world scenarios: (1) Noisy Reward Supervision, dependence on weak labels derived from post-hoc user actions introduces bias, particularly failing to capture satisfaction signals in ASR-error-induced utterances; (2) Long-Tail Feedback Sparsity, the power-law distribution of user queries causes reward prediction accuracy to drop in low-frequency domains. The noise in the weak labels and a power-law distribution of user utterances results in that the model is hard to learn good representation of user utterances and sessions. To address these limitations, we propose two auxiliary tasks to improve the representation learning of user utterances and sessions that enhance user satisfaction prediction. The first one is a contrastive self-supervised learning task, which helps the model learn the representation of rare user utterances and identify ASR errors. The second one is a domain-intent classification task, which aids the model in learning the representation of user sessions from long-tailed domains and improving the model's performance on such domains. The proposed method is evaluated on DuerOS, demonstrating significant improvements in the accuracy of error recognition on rare user utterances and long-tailed domains.