mlsanityData to 7 Oct 2026

Aligning Large Language Models with Human: A Survey

arXiv 2307.12966 ↗First posted 24 Jul 2023cs.CLcited by 31

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect (hallucinated) information. Hence, aligning LLMs with human expectations has become an active area of interest within the research community. This survey presents a comprehensive overview of these alignment technologies, including the following aspects. (1) Data collection: the methods for effectively collecting high-quality instructions for LLM alignment, including the use of NLP benchmarks, human annotations, and leveraging strong LLMs. (2) Training methodologies: a detailed review of the prevailing training methods employed for LLM alignment. Our exploration encompasses Supervised Fine-tuning, both Online and Offline human preference training, along with parameter-efficient training mechanisms. (3) Model Evaluation: the methods for evaluating the effectiveness of these human-aligned LLMs, presenting a multifaceted approach towards their assessment. In conclusion, we collate and distill our findings, shedding light on several promising future research avenues in the field. This survey, therefore, serves as a valuable resource for anyone invested in understanding and advancing the alignment of LLMs to better suit human-oriented tasks and expectations. An associated GitHub link collecting the latest papers is available at https://github.com/GaryYufei/AlignLLMHumanSurvey.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 69 of its 109 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

69 of 109 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models19 Jun
  3. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Experts24 May
  4. 2023LIMA: Less Is More for Alignment18 May
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 63

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
38 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  11. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  12. 2023Instruction Tuning with GPT-4cited by 676
  13. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  14. 2023Textbooks Are All You Needcited by 561
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  16. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  17. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  18. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  19. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  20. 2023Scaling Data-Constrained Language Modelscited by 391
  21. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  22. 2023GPTScore: Evaluate as You Desirecited by 357
  23. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  24. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  25. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  26. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  27. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  28. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  29. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  30. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  31. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  32. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  33. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  34. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  35. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  36. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  37. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57
  38. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 4; 2026: 7.

31 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  4. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  5. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  6. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Needcited by 22
  7. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  8. 2023Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoningcited by 19
  9. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  10. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Frameworkcited by 16
  11. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 16
  12. 2024Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agentscited by 15
19 more
  1. 2024Leveraging Large Language Models with Chain-of-Thought and Prompt Engineering for Traffic Crash Severity Analysis and Inferencecited by 14
  2. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 14
  3. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  4. 2024WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Taskscited by 11
  5. 2024Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuningcited by 10
  6. 2024Can Machines Think Like Humans? A Behavioral Evaluation of LLM Agents in Dictator Gamescited by 9
  7. 2024Should agentic conversational AI change how we think about ethics? Characterising an interactional ethics centred on respectcited by 8
  8. 2024Practical Considerations for Agentic LLM Systemscited by 8
  9. 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learningcited by 8
  10. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Modelscited by 7
  11. 2026Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancycited by 7
  12. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  13. 2024Self-training Language Models for Arithmetic Reasoningcited by 2
  14. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Modelscited by 1
  15. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  16. 2024Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluationcited by 0
  17. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoningcited by 0
  18. 2026LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routingcited by 0
  19. 2026LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2023Large Language Model Alignment: A Survey
  3. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples
  4. 2024Towards Scalable Automated Alignment of LLMs: A Survey
  5. 2025Aligning Multimodal LLM with Human Preference: A Survey
  6. 2025A Survey on Training-free Alignment of Large Language Models
  7. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
  8. 2024Aligners: Decoupling LLMs and Alignment