mlsanityData to 7 Oct 2026

Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback

arXiv 2303.02891 ↗First posted 6 Mar 2023cs.CY cs.AI cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Is it possible for machines to think like humans? And if it is, how should we go about teaching them to do so? As early as 1950, Alan Turing stated that we ought to teach machines in the way of teaching a child. Reinforcement learning with human feedback (RLHF) has emerged as a strong candidate toward allowing agents to learn from human feedback in a naturalistic manner. RLHF is distinct from traditional reinforcement learning as it provides feedback from a human teacher in addition to a reward signal. It has been catapulted into public view by multiple high-profile AI applications, including OpenAI's ChatGPT, DeepMind's Sparrow, and Anthropic's Claude. These highly capable chatbots are already overturning our understanding of how AI interacts with humanity. The wide applicability and burgeoning success of RLHF strongly motivate the need to evaluate its social impacts. In light of recent developments, this paper considers an important question: can RLHF be developed and used without negatively affecting human societies? Our objectives are threefold: to provide a systematic study of the social effects of RLHF; to identify key social and ethical issues of RLHF; and to discuss social impacts for stakeholders. Although text-based applications of RLHF have received much attention, it is crucial to consider when evaluating its social implications the diverse range of areas to which it may be deployed. We describe seven primary ways in which RLHF-based technologies will affect society by positively transforming human experiences with AI. This paper ultimately proposes that RLHF has potential to net positively impact areas of misinformation, AI value-alignment, bias, AI access, cross-cultural dialogue, industry, and workforce. As RLHF raises concerns that echo those of existing AI technologies, it will be important for all to be aware and intentional in the adoption of RLHF.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 39 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

15 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback21 Nov
  2. 2022The Expertise Problem: Learning from Specialized Feedback12 Nov
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  5. 2020Artificial Intelligence, Values and Alignmentcited by 443
  6. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  7. 2021Alignment of Language Agentscited by 167
  8. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  9. 2020Reinforcement learning with human advice: a surveycited by 23
  10. 2020Reinforcement Learning Under Moral Uncertaintycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 5; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practicecited by 86
  2. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  3. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  4. 2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactionscited by 50
  5. 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
  6. 2023Interactive Natural Language Processingcited by 31
  7. 2023Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulationscited by 11
  8. 2023AGI for Agriculturecited by 9
  9. 2023Fine-tuning Language Models with Generative Adversarial Reward Modellingcited by 8
  10. 2023BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?cited by 7
  11. 2024Beyond Accuracy Optimization: Computer Vision Losses for Large Language Model Fine-Tuningcited by 5
  12. 2024FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitationcited by 1
3 more
  1. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  2. 2025Mitigation of Gender and Ethnicity Bias in AI-Generated Stories through Model Explanationscited by 1
  3. 2026Surrogate modeling for interpreting black-box LLMs in medical predictionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  2. 2023A Survey of Reinforcement Learning from Human Feedback
  3. 2023The History and Risks of Reinforcement Learning and Human Feedback
  4. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  5. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  8. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback