mlsanityData to 7 Oct 2026

AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations

arXiv 2406.18346 ↗First posted 26 Jun 2024cs.AIcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

This paper critically evaluates the attempts to align Artificial Intelligence (AI) systems, especially Large Language Models (LLMs), with human values and intentions through Reinforcement Learning from Feedback (RLxF) methods, involving either human feedback (RLHF) or AI feedback (RLAIF). Specifically, we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. Through a multidisciplinary sociotechnical critique, we examine both the theoretical underpinnings and practical implementations of RLxF techniques, revealing significant limitations in their approach to capturing the complexities of human ethics and contributing to AI safety. We highlight tensions and contradictions inherent in the goals of RLxF. In addition, we discuss ethically-relevant issues that tend to be neglected in discussions about alignment and RLxF, among which the trade-offs between user-friendliness and deception, flexibility and interpretability, and system safety. We conclude by urging researchers and practitioners alike to critically assess the sociotechnical ramifications of RLxF, advocating for a more nuanced and reflective approach to its application in AI development.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 57 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

35 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
20 more
  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  3. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  4. 2019Assessing BERT's Syntactic Abilitiescited by 191
  5. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  6. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  7. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  8. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  9. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  10. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  11. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  12. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
  13. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 26
  14. 2023Concrete Problems in AI Safety, Revisitedcited by 21
  15. 2022System Safety and Artificial Intelligencecited by 18
  16. 2022The Data-Production Dispositifcited by 16
  17. 2023Algorithms as Social-Ecological-Technological Systems: an Environmental Justice Lens on Algorithmic Auditscited by 12
  18. 2023Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Developmentcited by 9
  19. 2023ACROCPoLis: A Descriptive Framework for Making Sense of Fairnesscited by 8
  20. 2022AI, Opacity, and Personal Autonomycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  2. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  3. 2026Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMscited by 2
  4. 2025Beyond Algorethics: Addressing the Ethical and Anthropological Challenges of AI Recommender Systemscited by 1
  5. 2026Mitigating loss of control in advanced AI systems through instrumental goal trajectoriescited by 1
  6. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0
  7. 2026Learning When to Trust in Contextual Social Banditscited by 0
  8. 2026ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Taskscited by 0
  9. 2026What Do People Actually Want From AI? Mapping Preference Pluralitycited by 0
  10. 2026Toward a Theory of Value in AI Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications
  2. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  4. 2023A Survey of Reinforcement Learning from Human Feedback
  5. 2024Position: Towards Bidirectional Human-AI Alignment
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  8. 2025RLTHF: Targeted Human Feedback for LLM Alignment