# Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators

arXiv: https://arxiv.org/abs/1904.13015 · HTML: https://mlsanity.com/paper/1904.13015/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2019.

## Built on by (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Chain of Hindsight Aligns Language Models with Feedback](https://mlsanity.com/paper/2302.02676/) (2023)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [Critic-Guided Decoding for Controlled Text Generation](https://mlsanity.com/paper/2212.10938/) (2022)
- [M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions](https://mlsanity.com/paper/2405.16420/) (2024)
- [A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice](https://mlsanity.com/paper/2404.16958/) (2024)
- [Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model](https://mlsanity.com/paper/2403.19443/) (2024)
- [A Survey on Human Preference Learning for Large Language Models](https://mlsanity.com/paper/2406.11191/) (2024)
- [Reward Modeling for Mitigating Toxicity in Transformer-based Language Models](https://mlsanity.com/paper/2202.09662/) (2022)
- [Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation](https://mlsanity.com/paper/2106.05894/) (2021)
- [RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks](https://mlsanity.com/paper/2511.01758/) (2025)
- [Towards Boosting the Open-Domain Chatbot with Human Feedback](https://mlsanity.com/paper/2208.14165/) (2022)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2402.04867/) (2024)
- [Human-in-the-loop Abstractive Dialogue Summarization](https://mlsanity.com/paper/2212.09750/) (2022)
- [Advancements in Scientific Controllable Text Generation Methods](https://mlsanity.com/paper/2307.05538/) (2023)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization](https://mlsanity.com/paper/2502.07237/) (2025)
- [Modeling Performance in Open-Domain Dialogue with PARADISE](https://mlsanity.com/paper/2110.11164/) (2021)
- [It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF](https://mlsanity.com/paper/2406.07971/) (2024)
- [A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts](https://mlsanity.com/paper/1907.05346/) (2019)
- [A Transformer-based Response Evaluator for Open-Domain Spoken Conversation](https://mlsanity.com/paper/2302.04424/) (2023)
- [A Novel Computational and Modeling Foundation for Automatic Coherence Assessment](https://mlsanity.com/paper/2310.00598/) (2023)
- [Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operations](https://mlsanity.com/paper/2106.11345/) (2021)
- [PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems](https://mlsanity.com/paper/2309.10413/) (2023)
- [Open-Domain Dialogue Quality Evaluation: Deriving Nugget-level Scores from Turn-level Scores](https://mlsanity.com/paper/2310.00410/) (2023)

## Abstract (arXiv)

Encoder-decoder based neural architectures serve as the basis of state-of-the-art approaches in end-to-end open domain dialog systems. Since most of such systems are trained with a maximum likelihood~(MLE) objective they suffer from issues such as lack of generalizability and the generic response problem, i.e., a system response that can be an answer to a large number of user utterances, e.g., "Maybe, I don't know." Having explicit feedback on the relevance and interestingness of a system response at each turn can be a useful signal for mitigating such issues and improving system quality by selecting responses from different approaches. Towards this goal, we present a system that evaluates chatbot responses at each dialog turn for coherence and engagement. Our system provides explicit turn-level dialog quality feedback, which we show to be highly correlated with human evaluation. To show that incorporating this feedback in the neural response generation models improves dialog quality, we present two different and complementary mechanisms to incorporate explicit feedback into a neural response generation model: reranking and direct modification of the loss function during training. Our studies show that a response generation model that incorporates these combined feedback mechanisms produce more engaging and coherent responses in an open-domain spoken dialog setting, significantly improving the response quality using both automatic and human evaluation.