# Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback

arXiv: https://arxiv.org/abs/2303.02891 · HTML: https://mlsanity.com/paper/2303.02891/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2211.11602/) (2022)
- [Reinforcement learning with human advice: a survey](https://mlsanity.com/paper/2005.11016/) (2020)
- [The Expertise Problem: Learning from Specialized Feedback](https://mlsanity.com/paper/2211.06519/) (2022)
- [Reinforcement Learning Under Moral Uncertainty](https://mlsanity.com/paper/2006.04734/) (2020)

## Built on by (most cited first, 15 of 15)

- [The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice](https://mlsanity.com/paper/2310.00907/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [Beyond the Safeguards: Exploring the Security Risks of ChatGPT](https://mlsanity.com/paper/2305.08005/) (2023)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations](https://mlsanity.com/paper/2311.06330/) (2023)
- [AGI for Agriculture](https://mlsanity.com/paper/2304.06136/) (2023)
- [Fine-tuning Language Models with Generative Adversarial Reward Modelling](https://mlsanity.com/paper/2305.06176/) (2023)
- [BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?](https://mlsanity.com/paper/2310.16681/) (2023)
- [Beyond Accuracy Optimization: Computer Vision Losses for Large Language Model Fine-Tuning](https://mlsanity.com/paper/2409.13641/) (2024)
- [FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitation](https://mlsanity.com/paper/2403.06267/) (2024)
- [Intuitions of Compromise: Utilitarianism vs. Contractualism](https://mlsanity.com/paper/2410.05496/) (2024)
- [Mitigation of Gender and Ethnicity Bias in AI-Generated Stories through Model Explanations](https://mlsanity.com/paper/2509.04515/) (2025)
- [Surrogate modeling for interpreting black-box LLMs in medical predictions](https://mlsanity.com/paper/2604.20331/) (2026)

## Abstract (arXiv)

Is it possible for machines to think like humans? And if it is, how should we go about teaching them to do so? As early as 1950, Alan Turing stated that we ought to teach machines in the way of teaching a child. Reinforcement learning with human feedback (RLHF) has emerged as a strong candidate toward allowing agents to learn from human feedback in a naturalistic manner. RLHF is distinct from traditional reinforcement learning as it provides feedback from a human teacher in addition to a reward signal. It has been catapulted into public view by multiple high-profile AI applications, including OpenAI's ChatGPT, DeepMind's Sparrow, and Anthropic's Claude. These highly capable chatbots are already overturning our understanding of how AI interacts with humanity. The wide applicability and burgeoning success of RLHF strongly motivate the need to evaluate its social impacts. In light of recent developments, this paper considers an important question: can RLHF be developed and used without negatively affecting human societies? Our objectives are threefold: to provide a systematic study of the social effects of RLHF; to identify key social and ethical issues of RLHF; and to discuss social impacts for stakeholders. Although text-based applications of RLHF have received much attention, it is crucial to consider when evaluating its social implications the diverse range of areas to which it may be deployed. We describe seven primary ways in which RLHF-based technologies will affect society by positively transforming human experiences with AI. This paper ultimately proposes that RLHF has potential to net positively impact areas of misinformation, AI value-alignment, bias, AI access, cross-cultural dialogue, industry, and workforce. As RLHF raises concerns that echo those of existing AI technologies, it will be important for all to be aware and intentional in the adoption of RLHF.