mlsanityData to 7 Oct 2026

Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines

arXiv 2605.26442 ↗First posted 7 Apr 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Much of the alignment tuning literature is organized around optimization objectives, while the construction of alignment data is often treated implicitly. In this survey, we adopt a data centric perspective and reframe alignment tuning as a pipeline design problem. We decompose alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing alignment methods into a unified taxonomy. Through this lens, we identify recurring design trade-offs and failure modes observed across prior alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal. Finally, we outline open challenges for alignment data pipelines, including prompt-level alignment, agentic settings, and alignment under evolving objectives.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 113 of its 123 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

113 of 123 references.

References · 113

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  13. 2024Large Language Models: A Surveycited by 506
  14. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  15. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  16. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  17. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  18. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  19. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  20. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  21. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  22. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  23. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  24. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  25. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  26. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  27. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  28. 2023AI Alignment: A Comprehensive Surveycited by 256
  29. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  30. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  31. 2024A Survey on Data Selection for Language Modelscited by 230
  32. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  33. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  34. 2023Mitigating the Alignment Tax of RLHFcited by 181
  35. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  36. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  37. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  38. 2024Token-level Direct Preference Optimizationcited by 135

50 most cited of 113 All 113 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Ready2Blend: From Natural-Language Instructions to Composable Alignment Promptscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AlignTune: Modular Toolkit for Post-Training Alignment of Large Language Models
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
  3. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  4. 2024Aligner: Efficient Alignment by Learning to Correct
  5. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2024Efficient Alignment of Large Language Models via Data Sampling
  8. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models