mlsanityData to 7 Oct 2026

Challenges and Future Directions of Data-Centric AI Alignment

arXiv 2410.01957 ↗First posted 2 Oct 2024cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As AI systems become increasingly capable and influential, ensuring their alignment with human values, preferences, and goals has become a critical research focus. Current alignment methods primarily focus on designing algorithms and loss functions but often underestimate the crucial role of data. This paper advocates for a shift towards data-centric AI alignment, emphasizing the need to enhance the quality and representativeness of data used in aligning AI systems. In this position paper, we highlight key challenges associated with both human-based and AI-based feedback within the data-centric alignment framework. Through qualitative analysis, we identify multiple sources of unreliability in human feedback, as well as problems related to temporal drift, context dependence, and AI-based feedback failing to capture human values due to inherent model limitations. We propose future research directions, including improved feedback collection practices, robust data-cleaning methodologies, and rigorous feedback verification processes. We call for future research into these critical directions to ensure, addressing gaps that persist in understanding and improving data-centric alignment practices.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 73 of its 96 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

73 of 96 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Improving Context-Aware Preference Modeling for Language Models20 Jul
  2. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
  3. 2024Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback9 Jun
  4. 2024AI Alignment with Changing and Influenceable Reward Functions28 May
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
  6. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  7. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition24 Feb
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  10. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  11. 2020Learning to summarize from human feedback2 Sep

Other references · 62

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2023Whose Opinions Do Language Models Reflect?cited by 759
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
38 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  6. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  7. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  8. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  9. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  10. 2021Unsolved Problems in ML Safetycited by 291
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  12. 2023AI Alignment: A Comprehensive Surveycited by 256
  13. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  14. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  15. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  16. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  17. 2023Nash Learning from Human Feedbackcited by 220
  18. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  19. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  20. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  21. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  22. 2023Model evaluation for extreme riskscited by 167
  23. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  24. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  25. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  26. 2024Token-level Direct Preference Optimizationcited by 135
  27. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  28. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  29. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  30. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  31. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  32. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  33. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  34. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  35. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  36. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  37. 2024Decoding-time Realignment of Language Modelscited by 67
  38. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024Process Reward Model with Q-Value Rankingscited by 78
  2. 2025Visual Instruction Bottleneck Tuningcited by 5
  3. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
  4. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  5. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Position: Towards Bidirectional Human-AI Alignment
  2. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  3. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  4. 2026Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
  5. 2025A Statistical Case Against Empirical Human-AI Alignment
  6. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
  7. 2026Building Intelligent User Interfaces for Human-AI Alignment
  8. 2026Align AI to Dynamic Human-AI Workflows