mlsanityData to 7 Oct 2026

Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation

arXiv 2402.14146 ↗First posted 21 Feb 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Textual style expresses a diverse set of information, including interpersonal dynamics (e.g., formality) and the author's emotions or attitudes (e.g., disgust). An open question is how language models can be explicitly controlled so that they weave together target styles when generating text: for example, to produce text that is both negative and non-toxic. One approach to such controlled generation is multi-objective reinforcement learning (RL), but how best to combine multiple objectives in a reward function is an open question. In this paper, we investigate various formulations of multi-style rewards, including calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes. We find that our proposed dynamic weighting outperforms static weighting approaches with respect to style control while maintaining linguistic quality, and we explore its effectiveness in 2- and 3-style control.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

26 of 42 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  8. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  9. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  10. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  12. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
14 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  3. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  4. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
  5. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  6. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  7. 2019Reinforcement Learning Based Text Style Transfer without Parallel Training Corpuscited by 38
  8. 2021Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliabilitycited by 30
  9. 2021Attribute Alignment: Controlling Text Generation from Pre-trained Language Modelscited by 27
  10. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 21
  11. 2022Efficient Reinforcement Learning for Unsupervised Controlled Text Generationcited by 7
  12. 2022On the Impact of Noises in Crowd-Sourced Data for Speech Translationcited by 6
  13. 2022Inverse Reinforcement Learning for Text Summarizationcited by 5
  14. 2023Balancing Effect of Training Dataset Distribution of Multiple Styles for Multi-Style Text Transfercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  2. 2025How Many Instructions Can LLMs Follow at Once?cited by 17
  3. 2025AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMscited by 8
  4. 2024LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedbackcited by 7
  5. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6
  6. 2026Steering Language Models Before They Speak: Logit-Level Interventionscited by 3
  7. 2026DLM-SWAI: Steering Diffusion Language Models Before They Unmaskcited by 2
  8. 2025Stealing Creator's Workflow: A Creator-Inspired Agentic Framework with Iterative Feedback Loop for Improved Scientific Short-form Generationcited by 1
  9. 2026Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuningcited by 1
  10. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptationcited by 1
  11. 2026Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RLcited by 1
  12. 2025A Hybrid Ai Framework For Strategic Patent Portfolio Pruning: Integrating Learning To-Rank And Market Need Analysis For Technology Transfer Optimizationcited by 0
1 more
  1. 2026Pareto-Guided Optimal Transport for Multi-Reward Alignmentcited by 0