mlsanityData to 7 Oct 2026

A General Language Assistant as a Laboratory for Alignment

arXiv 2112.00861 ↗First posted 1 Dec 2021cs.CL cs.LGcited by 1,023

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

Key facts. Cited by 1,023 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 60 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

36 of 60 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
24 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2018AI safety via debatecited by 384
  11. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  13. 2021Unsolved Problems in ML Safetycited by 291
  14. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  15. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  16. 2021Internet-Augmented Dialogue Generationcited by 217
  17. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  18. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  19. 2021Challenges in Detoxifying Language Modelscited by 169
  20. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  21. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  22. 2018Supervising strong learners by amplifying weak expertscited by 139
  23. 2021Scaling Scaling Laws with Board Gamescited by 62
  24. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 43; 2023: 203; 2024: 333; 2025: 244; 2026: 199.

1,023 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
38 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  13. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  14. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  15. 2023StarCoder: may the source be with you!cited by 790
  16. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  17. 2023Whose Opinions Do Language Models Reflect?cited by 759
  18. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  19. 2023Adversarial Diffusion Distillationcited by 741
  20. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  21. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  22. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  23. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  24. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  25. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  26. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  27. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  28. 2023Instruction Tuning with GPT-4cited by 676
  29. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  30. 2023A Comprehensive Overview of Large Language Modelscited by 639
  31. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  32. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  33. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  34. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  35. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  36. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  37. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  38. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454

50 most cited of 1,023 All 1,023 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  2. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  3. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  4. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2023Preference Ranking Optimization for Human Alignment
  7. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  8. 2024Uncovering Factor Level Preferences to Improve Human-Model Alignment