mlsanityData to 7 Oct 2026

UltraFeedback: Boosting Language Models with Scaled AI Feedback

arXiv 2310.01377 ↗First posted 2 Oct 2023cs.CL cs.AI cs.LGcited by 394

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Learning from human feedback has become a pivot technique in aligning large language models (LLMs) with human preferences. However, acquiring vast and premium human feedback is bottlenecked by time, labor, and human capability, resulting in small sizes or limited topics of current datasets. This further hinders feedback learning as well as alignment research within the open-source community. To address this issue, we explore how to go beyond human feedback and collect high-quality textitAI feedback automatically for a scalable alternative. Specifically, we identify scale and diversity as the key factors for feedback data to take effect. Accordingly, we first broaden instructions and responses in both amount and breadth to encompass a wider range of user-assistant interactions. Then, we meticulously apply a series of techniques to mitigate annotation biases for more reliable AI feedback. We finally present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset, which contains over 1 million GPT-4 feedback for 250k user-assistant conversations from various aspects. Built upon textscUltraFeedback, we align a LLaMA-based model by best-of-n sampling and reinforcement learning, demonstrating its exceptional performance on chat benchmarks. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models, serving as a solid foundation for future feedback learning research. Our data and models are available at https://github.com/thunlp/UltraFeedback.

Key facts. Cited by 394 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 63 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

49 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2020Learning to summarize from human feedback2 Sep

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
34 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  15. 2022Red Teaming Language Models with Language Modelscited by 983
  16. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  17. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  18. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  19. 2023Large Language Modelscited by 774
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  22. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  23. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  24. 2018AI safety via debatecited by 384
  25. 2022Self-critiquing models for assisting human evaluatorscited by 372
  26. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  27. 2023Tool Learning with Foundation Modelscited by 302
  28. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  29. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  30. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  31. 2018Supervising strong learners by amplifying weak expertscited by 139
  32. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  33. 2023Shepherd: A Critic for Language Model Generationcited by 96
  34. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 59; 2025: 186; 2026: 149.

394 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025Process Reinforcement through Implicit Rewardscited by 478
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2024Generative Representational Instruction Tuningcited by 227
  5. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  6. 2024Generative Reward Modelscited by 148
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  8. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  9. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  10. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  11. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 115
  12. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
38 more
  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  2. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  3. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  4. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  5. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  6. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  7. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  8. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  9. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  10. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  11. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
  12. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  13. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  14. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  15. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  16. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  17. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  18. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  19. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  20. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  21. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  22. 2024Reward-Robust RLHF in LLMscited by 32
  23. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  24. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  25. 2025BitNet b1.58 2B4T Technical Reportcited by 30
  26. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  27. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  28. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  29. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  30. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  31. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  32. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  33. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  34. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  35. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  36. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  37. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  38. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23

50 most cited of 394 All 394 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  2. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  3. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment
  5. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
  6. 2023Training Language Models with Language Feedback at Scale
  7. 2024Naturally Occurring Feedback is Common, Extractable and Useful
  8. 2022Training Language Models with Language Feedback