mlsanityData to 14 Oct 2026

Online Learning from Strategic Human Feedback in LLM Fine-Tuning

arXiv 2412.16834 ↗First posted 22 Dec 2024cs.AI cs.GTcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences. However, human labelers are selfish and have diverse preferences. They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences. Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret O(T) for T time slots. To our best knowledge, we are the first to study online learning mechanisms against strategic human labelers in the LLM fine-tuning process. We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation, ensuring their truthful feedback and sublinear regret O(T1/2). Simulation results demonstrate our mechanism's great advantages over the existing benchmark schemes.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 31 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

24 of 31 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  2. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  4. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  5. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  6. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  7. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  8. 2024Truthful Aggregation of LLMs with an Application to Online Advertisingcited by 29
  9. 2024Auctions with LLM Summariescited by 24
  10. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  11. 2020No-Regret and Incentive-Compatible Online Learningcited by 14
  12. 2017Online Prediction with Selfish Expertscited by 10
4 more
  1. 2024OPTune: Efficient Online Preference Tuningcited by 10
  2. 2021Efficient Competitions and Online Learning with Strategic Forecasterscited by 9
  3. 2023Mechanism Design for Collaborative Normal Mean Estimationcited by 7
  4. 2022Collaborative Algorithms for Online Personalized Mean Estimationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 2.

10 papers.

  1. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  2. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  3. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 4
  4. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  5. 2025The Battling Influencers Game: Nash Equilibria Structure of a Potential Game and Implications to Value Alignmentcited by 1
  6. 2025Incentivizing High-Quality Human Annotations with Golden Questionscited by 0
  7. 2025To Theoretically Understand Transformer-Based In-Context Learning for Optimizing CSMAcited by 0
  8. 2025Users as Annotators: LLM Preference Learning from Comparison Modecited by 0
  9. 2026Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcingcited by 0
  10. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
  4. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
  5. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  8. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning