# Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation

arXiv: https://arxiv.org/abs/2402.14146 · HTML: https://mlsanity.com/paper/2402.14146/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Controllable Natural Language Generation with Contrastive Prefixes](https://mlsanity.com/paper/2202.13257/) (2022)
- [Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus](https://mlsanity.com/paper/1903.10671/) (2019)
- [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021)
- [Attribute Alignment: Controlling Text Generation from Pre-trained Language Models](https://mlsanity.com/paper/2103.11070/) (2021)
- [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://mlsanity.com/paper/2210.01970/) (2022)
- [Efficient Reinforcement Learning for Unsupervised Controlled Text Generation](https://mlsanity.com/paper/2204.07696/) (2022)
- [On the Impact of Noises in Crowd-Sourced Data for Speech Translation](https://mlsanity.com/paper/2206.13756/) (2022)
- [Inverse Reinforcement Learning for Text Summarization](https://mlsanity.com/paper/2212.09917/) (2022)
- [Balancing Effect of Training Dataset Distribution of Multiple Styles for Multi-Style Text Transfer](https://mlsanity.com/paper/2305.15582/) (2023)

## Built on by (most cited first, 13 of 13)

- [RewardAnything: Generalizable Principle-Following Reward Models](https://mlsanity.com/paper/2506.03637/) (2025)
- [How Many Instructions Can LLMs Follow at Once?](https://mlsanity.com/paper/2507.11538/) (2025)
- [AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs](https://mlsanity.com/paper/2506.00569/) (2025)
- [LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback](https://mlsanity.com/paper/2406.03363/) (2024)
- [2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision](https://mlsanity.com/paper/2410.19720/) (2024)
- [Steering Language Models Before They Speak: Logit-Level Interventions](https://mlsanity.com/paper/2601.10960/) (2026)
- [Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning](https://mlsanity.com/paper/2604.05756/) (2026)
- [DLM-SWAI: Steering Diffusion Language Models Before They Unmask](https://mlsanity.com/paper/2605.29626/) (2026)
- [Stealing Creator's Workflow: A Creator-Inspired Agentic Framework with Iterative Feedback Loop for Improved Scientific Short-form Generation](https://mlsanity.com/paper/2504.18805/) (2025)
- [DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation](https://mlsanity.com/paper/2607.07669/) (2026)
- [Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL](https://mlsanity.com/paper/2607.29246/) (2026)
- [A Hybrid Ai Framework For Strategic Patent Portfolio Pruning: Integrating Learning To-Rank And Market Need Analysis For Technology Transfer Optimization](https://mlsanity.com/paper/2509.00958/) (2025)
- [Pareto-Guided Optimal Transport for Multi-Reward Alignment](https://mlsanity.com/paper/2605.13155/) (2026)

## Abstract (arXiv)

Textual style expresses a diverse set of information, including interpersonal dynamics (e.g., formality) and the author's emotions or attitudes (e.g., disgust). An open question is how language models can be explicitly controlled so that they weave together target styles when generating text: for example, to produce text that is both negative and non-toxic. One approach to such controlled generation is multi-objective reinforcement learning (RL), but how best to combine multiple objectives in a reward function is an open question. In this paper, we investigate various formulations of multi-style rewards, including calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes. We find that our proposed dynamic weighting outperforms static weighting approaches with respect to style control while maintaining linguistic quality, and we explore its effectiveness in 2- and 3-style control.