mlsanityData to 7 Oct 2026

SEAL: Systematic Error Analysis for Value ALignment

arXiv 2408.10270 ↗First posted 16 Aug 2024cs.LG cs.AI cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) aims to align language models (LMs) with human values by training reward models (RMs) on binary preferences and using these RMs to fine-tune the base LMs. Despite its importance, the internal mechanisms of RLHF remain poorly understood. This paper introduces new metrics to evaluate the effectiveness of modeling and aligning human values, namely feature imprint, alignment resistance and alignment robustness. We categorize alignment datasets into target features (desired values) and spoiler features (undesired concepts). By regressing RM scores against these features, we quantify the extent to which RMs reward them - a metric we term feature imprint. We define alignment resistance as the proportion of the preference dataset where RMs fail to match human preferences, and we assess alignment robustness by analyzing RM responses to perturbed inputs. Our experiments, utilizing open-source components like the Anthropic/hh-rlhf preference dataset and OpenAssistant RMs, reveal significant imprints of target features and a notable sensitivity to spoiler features. We observed a 26% incidence of alignment resistance in portions of the dataset where LM-labelers disagreed with human preferences. Furthermore, we find that misalignment often arises from ambiguous entries within the alignment dataset. These findings underscore the importance of scrutinizing both RMs and alignment datasets for a deeper understanding of value alignment.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 32 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

25 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
11 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  5. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  6. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  7. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  8. 2023Human Feedback is not Gold Standardcited by 82
  9. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  10. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  11. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  3. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  4. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  4. 2025Doubly Robust Alignment for Large Language Models
  5. 2025Aligning to What? Limits to RLHF Based Alignment
  6. 2025Alignment is Localized: A Causal Probe into Preference Layers
  7. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  8. 2023Aligning Large Language Models with Human Preferences through Representation Engineering