Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Reinforcement learning from human feedback (RLHF) has become a cornerstone for aligning large language models with human preferences. However, the heterogeneity of human feedback, driven by diverse individual contexts and preferences, poses significant challenges for reward learning. To address this, we propose a Low-rank Contextual RLHF (LoCo-RLHF) framework that integrates contextual information to better model heterogeneous feedback while maintaining computational efficiency. Our approach builds on a contextual preference model, leveraging the intrinsic low-rank structure of the interaction between user contexts and query-answer pairs to mitigate the high dimensionality of feature representations. Furthermore, we address the challenge of distributional shifts in feedback through our Pessimism in Reduced Subspace (PRS) policy, inspired by pessimistic offline reinforcement learning techniques. We theoretically demonstrate that our policy achieves a tighter sub-optimality gap compared to existing methods. Extensive experiments validate the effectiveness of LoCo-RLHF, showcasing its superior performance in personalized RLHF settings and its robustness to distribution shifts.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 65 references that are in the map. First posted on arXiv 27 Dec 2024.
Builds on
38 of 65 references.
Closest ancestors · 6
Same direction, newest first
- 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
- 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
- 2019Fine-Tuning Language Models from Human Preferences18 Sep
- 2017Deep reinforcement learning from human preferences12 Jun
Other references · 32
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
- 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
- 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
- 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 160
20 more
- 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
- 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
- 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
- 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
- 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
- 2019Bilinear Bandits with Low-rank Structurecited by 55
- 2016Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descentcited by 39
- 2019Inference and Uncertainty Quantification for Noisy Matrix Completioncited by 39
- 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
- 2017Stochastic Low-Rank Banditscited by 28
- 2020Bridging Convex and Nonconvex Optimization in Robust PCA: Noise, Outliers, and Missing Datacited by 23
- 2023Moral Machine or Tyranny of the Majority?cited by 22
- 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
- 2019Statistical Inferences of Linear Forms for Noisy Matrix Completioncited by 18
- 2022Uncertainty Quantification of MLE for Entity Ranking with Covariatescited by 16
- 2019Learning Markov models via low-rank optimizationcited by 12
- 2023Doubly High-Dimensional Contextual Bandits: An Interpretable Model for Joint Assortment-Pricingcited by 8
- 2024Covariate Assisted Entity Ranking with Sparse Intrinsic Scorescited by 7
- 2024Online Policy Learning and Inference by Matrix Completioncited by 3
- 2024Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimationcited by 1
Built on by
Citing papers per year: 2025: 4; 2026: 6.
10 papers.
- 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
- 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
- 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedbackcited by 9
- 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truthcited by 7
- 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
- 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
- 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
- 2026Perturbation is All You Need for Extrapolating Language Modelscited by 2
- 2026Efficient Online Lexicographic Generalized Low-Rank Matrix Banditscited by 2
- 2026Efficient Online Lexicographic Generalized Low-Rank Matrix Bandits5 Aug 2026
- 2026Perturbation is All You Need for Extrapolating Language Models5 May 2026
- 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning30 Apr 2026
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspective2 Apr 2026
- 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic1 Mar 2026
- 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth29 Jan 2026
- 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback2 Dec 2025
- 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking15 Oct 2025
- 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlap30 Sep 2025
- 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning3 Apr 2025
Similar, not cited
Close in wording, no citation either way
- 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
- 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
- 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
- 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
- 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
- 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
- 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
- 2024Parameter Efficient Reinforcement Learning from Human Feedback