mlsanityData to 13 Oct 2026

Training language models to follow instructions with human feedback

arXiv 2203.02155 ↗First posted 4 Mar 2022cs.CL cs.AI cs.LGcited by 15,856

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

Key facts. Cited by 15,856 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 69 of its 83 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

69 of 83 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 68

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  6. 2017Deep reinforcement learning from human preferencescited by 3,871
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  10. 2020Learning to summarize from human feedbackcited by 2,575
  11. 2020Extracting Training Data from Large Language Modelscited by 2,181
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
38 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,736
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,546
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,249
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  9. 2021Ethical and social risks of harm from Language Modelscited by 964
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  11. 2017Constrained Policy Optimizationcited by 931
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 895
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 890
  14. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  15. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 797
  16. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 757
  17. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 728
  18. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 705
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 697
  20. 2019Release Strategies and the Social Impacts of Language Modelscited by 511
  21. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  22. 2018Scalable agent alignment via reward modeling: a research directioncited by 443
  23. 2020Artificial Intelligence, Values and Alignmentcited by 443
  24. 2018Gender Bias in Coreference Resolutioncited by 439
  25. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 411
  26. 2018QuAC : Question Answering in Contextcited by 380
  27. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  28. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  29. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  30. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  31. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  32. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 246
  33. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  34. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  35. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  36. 2021Alignment of Language Agentscited by 168
  37. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  38. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 299; 2023: 3003; 2024: 4233; 2025: 4291; 2026: 4030.

15,856 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2023Visual Instruction Tuningcited by 7,827
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  12. 2023Let's Verify Step by Stepcited by 3,672
38 more
  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,541
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
  5. 2023Qwen Technical Reportcited by 3,310
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,729
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,604
  13. 2023A Survey of Large Language Modelscited by 2,559
  14. 2022Emergent Abilities of Large Language Modelscited by 2,499
  15. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,344
  16. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  17. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  18. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
  19. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  20. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,118
  21. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,003
  22. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,953
  23. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  24. 2022Large Language Models Encode Clinical Knowledgecited by 1,817
  25. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,815
  26. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,799
  27. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  28. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,758
  29. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  30. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,637
  31. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,594
  32. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,450
  33. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  34. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,417
  35. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  36. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  37. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,399
  38. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,392

50 most cited of 15,856 All 15,856 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Teaching Language Models to Self-Improve by Learning from Language Feedback
  2. 2024ICE-GRT: Instruction Context Enhancement by Generative Reinforcement based Transformers
  3. 2025Aligning Instruction Tuning with Pre-training
  4. 2024Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data
  5. 2024Policy Improvement using Language Feedback Models
  6. 2024CogBench: a large language model walks into a psychology lab
  7. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  8. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility