mlsanityData to 7 Oct 2026

Understanding Impact of Human Feedback via Influence Functions

arXiv 2501.05790 ↗First posted 10 Jan 2025cs.AI cs.HC cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback to align large language models (LLMs) with human intentions. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses. Such feedback can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models. We propose a compute-efficient approximation method that enables the application of influence functions to LLM-based reward models and large-scale preference datasets. Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases in human feedback datasets and (2) guiding labelers in refining their strategies to better align with expert feedback. By quantifying the impact of human feedback, we believe that influence functions can enhance feedback interpretability and contribute to scalable oversight in RLHF, helping labelers provide more accurate and consistent feedback. Source code is available at https://github.com/mintaywon/IF_RLHF

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 49 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

30 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
13 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  3. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  6. 2022Self-critiquing models for assisting human evaluatorscited by 372
  7. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  8. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  9. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  10. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  11. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  12. 2019Detecting Adversarial Samples Using Influence Functions and Nearest Neighborscited by 72
  13. 2023OPORP: One Permutation + One Random Projectioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025Large Language Model Sourcing: A Surveycited by 5
  2. 2025Sycophancy Claims about Language Models: The Missing Human-in-the-Loopcited by 4
  3. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  4. 2025Efficient Multi-Agent System Training with Data Influence-Oriented Tree Searchcited by 3
  5. 2025Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attributioncited by 3
  6. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 3
  7. 2025Newfluence: Boosting Model interpretability and Understanding in High Dimensionscited by 2
  8. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  9. 2026Quality over Quantity: Demonstration Curation via Influence Functions for Data-Centric Robot Learningcited by 1
  10. 2026Refining Multidimensional Video Reward Models via Disentangled Influence Functionscited by 0
  11. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Influencing Humans to Conform to Preference Models for RLHF
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  4. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  5. 2025Influence Functions for Preference Dataset Pruning
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2026Mitigating Cognitive Bias in RLHF by Altering Rationality
  8. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback