mlsanityData to 7 Oct 2026

Hummer: Towards Limited Competitive Preference Dataset

arXiv 2405.11647 ↗First posted 19 May 2024cs.AI cs.LGcited by 10

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present textttHummer and its fine-grained variant, textttHummer-F, as innovative pairwise preference datasets with reduced-conflict alignment objectives. textttHummer is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions HummerRM as an ideal model for domain-specific further fine-tuning and reducing vulnerabilities to attacks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 69 of its 86 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

69 of 86 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  2. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  3. 2024Direct Preference Optimization with an Offset16 Feb
  4. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  5. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  6. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment7 Nov
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  9. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  10. 2023Preference Ranking Optimization for Human Alignment30 Jun
  11. 2022Training language models to follow instructions with human feedback4 Mar
  12. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 57

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
38 more
  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2023Llemma: An Open Language Model For Mathematicscited by 385
  12. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  13. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  14. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  15. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  16. 2023AI Alignment: A Comprehensive Surveycited by 256
  17. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  18. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  19. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  20. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  21. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  22. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  23. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  24. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  25. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  26. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  27. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  28. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  29. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  30. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  31. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  32. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  33. 2023Red Teaming Language Model Detectors with Language Modelscited by 55
  34. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  35. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  36. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  37. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  38. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 41
  2. 2024Adaptive Token Biaser: Knowledge Editing via Biasing Key Entitiescited by 14
  3. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  4. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  5. 2025Efficient Whole Slide Pathology VQA via Token Compressioncited by 6
  6. 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasetscited by 5
  7. 2025SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learningcited by 5
  8. 2026Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructurecited by 3
  9. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
  10. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison
  2. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  3. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
  4. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  5. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2025HPS: Hard Preference Sampling for Human Preference Alignment
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment