mlsanityData to 7 Oct 2026

The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback

arXiv 2405.11226 ↗First posted 18 May 2024cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has contributed to performance improvements in large language models. To tackle its reliance on substantial amounts of human-labeled data, a successful approach is multi-task representation learning, which involves learning a high-quality, low-dimensional representation from a wide range of source tasks. In this paper, we formulate RLHF as the contextual dueling bandit problem and assume a common linear representation. We demonstrate that the sample complexity of source tasks in multi-task RLHF can be reduced by considering task relevance and allocating different sample sizes to source tasks with varying task relevance. We further propose an algorithm to estimate task relevance by a small number of additional data and then learn a policy. We prove that to achieve varepsilon-optimal, the sample complexity of the source tasks can be significantly reduced compared to uniform sampling. Additionally, the sample complexity of the target task is only linear in the dimension of the latent space, thanks to representation learning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 58 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

39 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  2. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  7. 2013Learning to Optimize Via Posterior Samplingcited by 253
  8. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  9. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  10. 2015The Benefit of Multitask Representation Learningcited by 194
  11. 2020Provable Meta-Learning of Linear Representationscited by 161
  12. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
20 more
  1. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  2. 2017Scalable Generalized Linear Bandits: Online Computation and Hashingcited by 63
  3. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  4. 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
  5. 2020Instance-Wise Minimax-Optimal Algorithms for Logistic Banditscited by 56
  6. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  7. 2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
  8. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  9. 2024Sharing Knowledge in Multi-Task Deep Reinforcement Learningcited by 42
  10. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  11. 2018An Information-Theoretic Analysis for Thompson Sampling with Many Actionscited by 35
  12. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  13. 2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
  14. 2018Active Ranking with Subset-wise Preferencescited by 21
  15. 2018PG-TS: Improved Thompson Sampling for Logistic Contextual Banditscited by 18
  16. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  17. 2022Active Multi-Task Representation Learningcited by 13
  18. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
  19. 2022Exploiting Correlation to Achieve Faster Learning Rates in Low-Rank Preference Banditscited by 11
  20. 2023Improved Active Multi-Task Representation Learning via Lassocited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Textscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learning
  2. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling
  3. 2025Multi-Task Reward Learning from Human Ratings
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
  6. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  7. 2025Reinforcement Learning from Human Feedback
  8. 2025Generalisation in Multitask Fitted Q-Iteration and Offline Q-learning