mlsanityData to 7 Oct 2026

Aligning to What? Limits to RLHF Based Alignment

arXiv 2503.09025 ↗First posted 12 Mar 2025cs.CLcited by 8

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is increasingly used to align large language models (LLMs) with human preferences. However, the effectiveness of RLHF in addressing underlying biases remains unclear. This study investigates the relationship between RLHF and both covert and overt biases in LLMs, particularly focusing on biases against African Americans. We applied various RLHF techniques (DPO, ORPO, and RLOO) to Llama 3 8B and evaluated the covert and overt biases of the resulting models using matched-guise probing and explicit bias testing. We performed additional tests with DPO on different base models and datasets; among several implications, we found that SFT before RLHF calcifies model biases. Additionally, we extend the tools for measuring biases to multi-modal models. Through our experiments we collect evidence that indicates that current alignment techniques are inadequate for nebulous tasks such as mitigating covert biases, highlighting the need for capable datasets, data curating techniques, or alignment tools.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 31 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

15 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  9. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  11. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  12. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 35
1 more
  1. 2017Bandit Structured Prediction for Neural Sequence-to-Sequence Learningcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  3. 2025Trustless Autonomy: Understanding Motivations, Benefits, and Governance Dilemmas in Self-Sovereign Decentralized AI Agentscited by 10
  4. 2026Analysing the Safety Pitfalls of Steering Vectorscited by 9
  5. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrongcited by 3
  6. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  7. 2026Greener Than Humans? Environmental Attitudes in Large Language Modelscited by 0
  8. 2026Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2024SEAL: Systematic Error Analysis for Value ALignment
  4. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  5. 2023Aligning Large Language Models with Human Preferences through Representation Engineering
  6. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
  7. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  8. 2026Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner