mlsanityData to 7 Oct 2026

Reward Dropout Improves Control: Bi-objective Perspective on Reinforced LM

arXiv 2310.04483 ↗First posted 6 Oct 2023cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the theoretical aspects of Reinforced Language Models (RLMs) from a bi-objective optimization perspective. Specifically, we consider the RLMs as a Pareto optimization problem that maximizes the two conflicting objectives, i.e., reward objective and likelihood objectives, simultaneously. Our main contribution consists of three parts. First, we establish the theoretical foundations of RLM as a Pareto optimization problem by presenting Reward Upper BOund (RUBO) and Pareto optimality. Our theoretical outcomes are supported by not only deductive proofs but also empirical results. Second, we propose Reward Dropout, a simple yet powerful method that guarantees to improve a bi-objective optimization of RLM. Lastly, we demonstrate that the Reward Dropout is consistently effective across five benchmark datasets and four benchmark LLMs, meaning that the Reward Dropout significantly improves the optimization performance of RLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

45 of 63 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  12. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
33 more
  1. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  4. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  5. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  6. 2017Toward Controlled Generation of Textcited by 271
  7. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 264
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  9. 2019Pareto Multi-Task Learningcited by 224
  10. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  11. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  12. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  13. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  14. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  15. 2017Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Modelscited by 116
  16. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  17. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  18. 2019Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representationcited by 84
  19. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  20. 2020Politeness Transfer: A Tag and Generate Approachcited by 78
  21. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  22. 2017Paraphrase Generation with Deep Reinforcement Learningcited by 58
  23. 2018Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approachcited by 55
  24. 2022Pareto Set Learning for Expensive Multi-Objective Optimizationcited by 51
  25. 2020Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generationcited by 47
  26. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  27. 2017Deep Reinforcement Learning for List-wise Recommendationscited by 35
  28. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  29. 2019Model-Based Reinforcement Learning with Adversarial Training for Online Recommendationcited by 33
  30. 2020On Learning Text Style Transfer with Direct Rewardscited by 24
  31. 2020RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learningcited by 19
  32. 2019Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shiftcited by 4
  33. 2022Bridging the Gap Between Training and Inference of Bayesian Controllable Language Modelscited by 1

Built on by

Not cited yet.