mlsanityData to 14 Oct 2026

Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation

arXiv 2403.13578 ↗First posted 20 Mar 2024cs.CL cs.LGcited by 10

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 48 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

26 of 48 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  6. 2016Self-critical Sequence Training for Image Captioningcited by 547
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  8. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  9. 2017Automated Curriculum Learning for Neural Networkscited by 257
  10. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  11. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  12. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 201
14 more
  1. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  2. 2021Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approachcited by 75
  3. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  4. 2018A Contextual Bandit Bake-offcited by 64
  5. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  6. 2021Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewardscited by 22
  7. 2022Why is constrained neural language generation particularly challenging?cited by 19
  8. 2015A Survey of Online Experiment Design with the Stochastic Multi-Armed Banditcited by 18
  9. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  10. 2023Boosting Distress Support Dialogue Responses with Motivational Interviewing Strategycited by 15
  11. 2021Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representationscited by 13
  12. 2018Adapting multi-armed bandits policies to contextual bandits scenarioscited by 8
  13. 2019Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generationcited by 6
  14. 2020DORB: Dynamically Optimizing Multiple Rewards with Banditscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 3.

10 papers.

  1. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  2. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  3. 2025A Fully Generative Motivational Interviewing Counsellor Chatbot for Moving Smokers Towards the Decision to Quitcited by 5
  4. 2024Prior Constraints-based Reward Model Training for Aligning Large Language Modelscited by 4
  5. 2025EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuningcited by 4
  6. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4
  7. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  8. 2025Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspectivecited by 1
  9. 2026One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systemscited by 1
  10. 2026Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  2. 2024Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation
  3. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  4. 2026RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation
  5. 2019Fine-Tuning Language Models from Human Preferences
  6. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  7. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  8. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria