mlsanityData to 7 Oct 2026

From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models

arXiv 2308.12014 ↗First posted 23 Aug 2023cs.AI cs.CL cs.CYcited by 45

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Big models, exemplified by Large Language Models (LLMs), are models typically pre-trained on massive data and comprised of enormous parameters, which not only obtain significantly improved performance across diverse tasks but also present emergent capabilities absent in smaller models. However, the growing intertwining of big models with everyday human lives poses potential risks and might cause serious social harm. Therefore, many efforts have been made to align LLMs with humans to make them better follow user instructions and satisfy human preferences. Nevertheless, `what to align with' has not been fully discussed, and inappropriate alignment goals might even backfire. In this paper, we conduct a comprehensive survey of different alignment goals in existing work and trace their evolution paths to help identify the most essential goal. Particularly, we investigate related works from two perspectives: the definition of alignment goals and alignment evaluation. Our analysis encompasses three distinct levels of alignment goals and reveals a goal transformation from fundamental abilities to value orientation, indicating the potential of intrinsic human values as the alignment goal for enhanced LLMs. Based on such results, we further discuss the challenges of achieving such intrinsic value alignment and provide a collection of available resources for future research on the alignment of big models.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 88 of its 110 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

88 of 110 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  2. 2023Training Language Models with Language Feedback at Scale28 Mar
  3. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2020Learning to summarize from human feedback2 Sep
  7. 2020Aligning AI With Shared Human Values5 Aug
  8. 2019Learning Norms from Stories: A Prior for Value Aligned Agents7 Dec

Other references · 80

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  14. 2023LIMA: Less Is More for Alignmentcited by 1,080
  15. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  16. 2021Ethical and social risks of harm from Language Modelscited by 957
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  18. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  19. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  22. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  23. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  24. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  25. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  26. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  27. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  28. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  29. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  30. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  31. 2020Artificial Intelligence, Values and Alignmentcited by 443
  32. 2018Gender Bias in Coreference Resolutioncited by 438
  33. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  34. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  35. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  36. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  37. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  38. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 17; 2025: 14; 2026: 6.

45 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  3. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  4. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  5. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  6. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  7. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  8. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  9. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  10. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 49
  11. 2024Towards Human-centered Proactive Conversational Agentscited by 48
  12. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
33 more
  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  2. 2023Probing the Moral Development of Large Language Models through Defining Issues Testcited by 23
  3. 2024DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subculturescited by 23
  4. 2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
  5. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  6. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  7. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 18
  8. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  9. 2024Agent Alignment in Evolving Social Normscited by 14
  10. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  11. 2024The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuningcited by 10
  12. 2023Unpacking the Ethical Value Alignment in Big Modelscited by 9
  13. 2024Do Moral Judgment and Reasoning Capability of LLMs Change with Language? A Study using the Multilingual Defining Issues Testcited by 9
  14. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  15. 2025Opportunities and Challenges of LLMs in Education: An NLP Perspectivecited by 6
  16. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  17. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  18. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
  19. 2025FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Modelscited by 5
  20. 2023Post Turing: Mapping the landscape of LLM Evaluationcited by 4
  21. 2025Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Modelscited by 4
  22. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  23. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  24. 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Makingcited by 4
  25. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
  26. 2026VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Modelscited by 3
  27. 2024Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Modelscited by 2
  28. 2025Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traitscited by 2
  29. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  30. 2025Contemplative Artificial Intelligencecited by 1
  31. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMscited by 1
  32. 2026Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmarkcited by 0
  33. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2025DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values
  3. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models
  4. 2023Large Language Model Alignment: A Survey
  5. 2023Heterogeneous Value Alignment Evaluation for Large Language Models
  6. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  7. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  8. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment