mlsanityData to 14 Oct 2026

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

arXiv 2305.14387 ↗First posted 22 May 2023cs.LG cs.AI cs.CLcited by 748

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their strong instruction-following abilities. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following requires tackling three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are 50x cheaper than crowdworkers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, DPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release all components of AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.

Key facts. Cited by 748 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 75 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

58 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Training Language Models with Language Feedback at Scale28 Mar
  2. 2023Pretraining Language Models with Human Preferences16 Feb
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  6. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct
  7. 2020Learning to summarize from human feedback2 Sep

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
38 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2018DeepMind Control Suitecited by 956
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  10. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  11. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  13. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  15. 2023Instruction Tuning with GPT-4cited by 686
  16. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  17. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  18. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  19. 2022Self-critiquing models for assisting human evaluatorscited by 380
  20. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  21. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  22. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  23. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  24. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  25. 2018Unity: A General Platform for Intelligent Agentscited by 236
  26. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  27. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  28. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  29. 2021TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generationcited by 120
  30. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  31. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 76
  32. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
  33. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72
  34. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  35. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 33
  36. 2016Dialog-based Language Learningcited by 21
  37. 2016Bandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translationcited by 8
  38. 2018A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translationcited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 116; 2024: 301; 2025: 206; 2026: 125.

748 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  5. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  8. 2024Self-Rewarding Language Modelscited by 681
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  10. 2023A Comprehensive Overview of Large Language Modelscited by 648
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  12. 2023Textbooks Are All You Needcited by 569
38 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  3. 2023The Falcon Series of Open Language Modelscited by 535
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  9. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  10. 2024LoRA Learns Less and Forgets Lesscited by 328
  11. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  12. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  13. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  14. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  15. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  16. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  17. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  18. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  19. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  20. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  21. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  22. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  23. 2023Large Language Model Alignment: A Surveycited by 243
  24. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  25. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  26. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  27. 2024Generative Representational Instruction Tuningcited by 233
  28. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  29. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  30. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  31. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  32. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  33. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  34. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  35. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  36. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  37. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
  38. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200

50 most cited of 748 All 748 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback
  2. 2024The Future of Open Human Feedback
  3. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  4. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
  5. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents
  6. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  8. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models