mlsanityData to 7 Oct 2026

Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression

arXiv 2501.12698 ↗First posted 22 Jan 2025cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

To improve user engagement during conversations with dialogue systems, we must improve individual dialogue responses and dialogue impressions such as consistency, personality, and empathy throughout the entire dialogue. While such dialogue systems have been developing rapidly with the help of large language models (LLMs), reinforcement learning from AI feedback (RLAIF) has attracted attention to align LLM-based dialogue models for such dialogue impressions. In RLAIF, a reward model based on another LLM is used to create a training signal for an LLM-based dialogue model using zero-shot/few-shot prompting techniques. However, evaluating an entire dialogue only by prompting LLMs is challenging. In this study, the supervised fine-tuning (SFT) of LLMs prepared reward models corresponding to 12 metrics related to the impression of the entire dialogue for evaluating dialogue responses. We tuned our dialogue models using the reward model signals as feedback to improve the impression of the system. The results of automatic and human evaluations showed that tuning the dialogue model using our reward model corresponding to dialogue impression improved the evaluation of individual metrics and the naturalness of the dialogue response.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 25 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

19 of 25 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2023Reward Design with Language Modelscited by 257
7 more
  1. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 177
  2. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  3. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  4. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  5. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  6. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  7. 2021Empirical Analysis of Training Strategies of Transformer-based Japanese Chit-chat Systemscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition
  2. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluators
  3. 2024Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback
  4. 2024Leveraging LLMs for Dialogue Quality Measurement
  5. 2020Human-centric Dialog Training via Offline Reinforcement Learning
  6. 2024Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
  7. 2024Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations
  8. 2021Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management