mlsanityData to 7 Oct 2026

Rewarding Chatbots for Real-World Engagement with Millions of Users

arXiv 2303.06135 ↗First posted 10 Mar 2023cs.CL cs.AI cs.LGcited by 58

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The emergence of pretrained large language models has led to the deployment of a range of social chatbots for chitchat. Although these chatbots demonstrate language ability and fluency, they are not guaranteed to be engaging and can struggle to retain users. This work investigates the development of social chatbots that prioritize user engagement to enhance retention, specifically examining the use of human feedback to efficiently develop highly engaging chatbots. The proposed approach uses automatic pseudo-labels collected from user interactions to train a reward model that can be used to reject low-scoring sample responses generated by the chatbot model at inference time. Intuitive evaluation metrics, such as mean conversation length (MCL), are introduced as proxies to measure the level of engagement of deployed chatbots. A/B testing on groups of 10,000 new daily chatbot users on the Chai Research platform shows that this approach increases the MCL by up to 70%, which translates to a more than 30% increase in user retention for a GPT-J 6B model. Future work aims to use the reward model to realise a data fly-wheel, where the latest user conversations can be used to alternately fine-tune the language model and the reward model.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 46 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

31 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  11. 2020Recipes for building an open-domain chatbotcited by 593
  12. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
17 more
  1. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  2. 2020Artificial Intelligence, Values and Alignmentcited by 443
  3. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  4. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  5. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  6. 2015A Neural Network Approach to Context-Sensitive Generation of Conversational Responsescited by 98
  7. 2018From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbotscited by 85
  8. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  9. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  10. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  11. 2020Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systemscited by 26
  12. 2021A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLPcited by 25
  13. 2017Evaluating Quality of Chatbots and Intelligent Conversational Agentscited by 16
  14. 2021UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialoguescited by 12
  15. 2019Ensemble-Based Deep Reinforcement Learning for Chatbotscited by 11
  16. 2022Grounding in social media: An approach to building a chit-chat dialogue modelcited by 4
  17. 2018Context-Aware Sequence-to-Sequence Models for Conversational Systemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 25; 2026: 11.

58 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  5. 2025Deep Think with Confidencecited by 199
  6. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  7. 2025S*: Test Time Scaling for Code Generationcited by 95
  8. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  9. 2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 64
  10. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  11. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  12. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
38 more
  1. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  2. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  3. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  4. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  5. 2024Taking AI Welfare Seriouslycited by 28
  6. 2024Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLMcited by 26
  7. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  8. 2024MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translationcited by 17
  9. 2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 16
  10. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  11. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 14
  12. 2025AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learningcited by 13
  13. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  14. 2025Learning to Refine: Self-Refinement of Parallel Reasoning in LLMscited by 12
  15. 2024PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluationcited by 10
  16. 2025From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignmentcited by 9
  17. 2025LatentEvolve: Self-Evolving Test-Time Scaling in Latent Spacecited by 8
  18. 2026UniT: Unified Multimodal Chain-of-Thought Test-time Scalingcited by 8
  19. 2024Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agentscited by 7
  20. 2026Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Modelscited by 7
  21. 2024A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Casecited by 6
  22. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignmentcited by 6
  23. 2025Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accuratelycited by 6
  24. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  25. 2026Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Modelcited by 6
  26. 2024Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Language Model Agentscited by 5
  27. 2025It's Not That Simple. An Analysis of Simple Test-Time Scalingcited by 5
  28. 2024Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversationscited by 4
  29. 2025CyclicReflex: Improving Reasoning Models via Cyclical Reflection Token Schedulingcited by 4
  30. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  31. 2026CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Computecited by 3
  32. 2023Safer Conversational AI as a Source of User Delightcited by 2
  33. 2026Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoEcited by 2
  34. 2023The Chai Platform's AI Safety Frameworkcited by 1
  35. 2024Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedbackcited by 1
  36. 2025Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factualitycited by 1
  37. 2025CLARIFID: Improving Radiology Report Generation by Reinforcing Clinically Accurate Impressions and Enforcing Detailed Findingscited by 1
  38. 2025FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized Collaborationcited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
  2. 2022Towards Boosting the Open-Domain Chatbot with Human Feedback
  3. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactions
  4. 2025Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments
  5. 2024Self-Evolved Reward Learning for LLMs
  6. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scores
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling
  8. 2025RewardBench 2: Advancing Reward Model Evaluation