# Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?

arXiv: https://arxiv.org/abs/2407.17482 · HTML: https://mlsanity.com/paper/2407.17482/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 16 of its 59 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [On the Impossible Safety of Large AI Models](https://mlsanity.com/paper/2209.15259/) (2022)
- [Towards a Benchmark for Scientific Understanding in Humans and Machines](https://mlsanity.com/paper/2304.10327/) (2023)
- [Targeting Solutions in Bayesian Multi-Objective Optimization: Sequential and Batch Versions](https://mlsanity.com/paper/1811.03862/) (2018)
- [Reinforcement Learning with Feedback from Multiple Humans with Diverse Skills](https://mlsanity.com/paper/2111.08596/) (2021)

## Built on by (most cited first, 8 of 8)

- [Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications](https://mlsanity.com/paper/2602.16201/) (2026)
- [Ethics and Persuasion in Reinforcement Learning from Human Feedback: A Procedural Rhetorical Approach](https://mlsanity.com/paper/2505.09576/) (2025)
- [PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs](https://mlsanity.com/paper/2509.20105/) (2025)
- [A Multifaceted Analysis of Social Biases in Large Language Models](https://mlsanity.com/paper/2512.15792/) (2025)
- [Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences](https://mlsanity.com/paper/2604.01312/) (2026)
- [Greener Than Humans? Environmental Attitudes in Large Language Models](https://mlsanity.com/paper/2606.02741/) (2026)
- [Beyond Epistemia: Epistemic Schizologia and Large Language Models as Techno-Semiotic Machines](https://mlsanity.com/paper/2607.25620/) (2026)
- [Controllable Affective Generation via Latent Vector Steering](https://mlsanity.com/paper/2608.25569/) (2026)

## Abstract (arXiv)

We argue for the epistemic and ethical advantages of pluralism in Reinforcement Learning from Human Feedback (RLHF) in the context of Large Language Models (LLM). Drawing on social epistemology and pluralist philosophy of science, we suggest ways in which RHLF can be made more responsive to human needs and how we can address challenges along the way. The paper concludes with an agenda for change, i.e. concrete, actionable steps to improve LLM development.