mlsanityData to 7 Oct 2026

Clone-Robust AI Alignment

arXiv 2501.09254 ↗First posted 16 Jan 2025cs.LG cs.AI cs.GTcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A key challenge in training Large Language Models (LLMs) is properly aligning them with human preferences. Reinforcement Learning with Human Feedback (RLHF) uses pairwise comparisons from human annotators to train reward functions and has emerged as a popular alignment method. However, input datasets in RLHF are not necessarily balanced in the types of questions and answers that are included. Therefore, we want RLHF algorithms to perform well even when the set of alternatives is not uniformly distributed. Drawing on insights from social choice theory, we introduce robustness to approximate clones, a desirable property of RLHF algorithms which requires that adding near-duplicate alternatives does not significantly change the learned reward function. We first demonstrate that the standard RLHF algorithm based on regularized maximum likelihood estimation (MLE) fails to satisfy this property. We then propose the weighted MLE, a new RLHF algorithm that modifies the standard regularized MLE by weighting alternatives based on their similarity to other alternatives. This new algorithm guarantees robustness to approximate clones while preserving desirable theoretical properties.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 27 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

19 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  4. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  5. 2024A Roadmap to Pluralistic Alignmentcited by 175
  6. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  7. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  8. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  9. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  10. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  11. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  12. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
6 more
  1. 2024Mapping Social Choice Theory to RLHFcited by 26
  2. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  3. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  4. 2022On the identifiability of mixtures of ranking modelscited by 7
  5. 2023RLHF and IIA: Perverse Incentivescited by 6
  6. 2022Obvious Independence of Clonescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  2. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  3. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  4. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  5. 2025Collaborating with GenAI: Incentives and Replacementscited by 1
  6. 2026The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisionscited by 0
  7. 2026AI Alignment From Social Choice Perspectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Doubly Robust Alignment for Large Language Models
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2025Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma
  4. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  5. 2025On a Connection Between Imitation Learning and RLHF
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  8. 2023Preference Ranking Optimization for Human Alignment