mlsanityData to 7 Oct 2026

WebGPT: Browser-assisted question-answering with human feedback

arXiv 2112.09332 ↗First posted 17 Dec 2021cs.CL cs.AI cs.LGcited by 1,722

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.

Key facts. Cited by 1,722 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 44 references that are in the map. First posted on arXiv 17 Dec 2021.

Builds on

22 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  10. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  11. 2019ELI5: Long Form Question Answeringcited by 567
  12. 2018AI safety via debatecited by 384
9 more
  1. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  2. 2018Supervising strong learners by amplifying weak expertscited by 139
  3. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  4. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  5. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  6. 2018Learning to Navigate the Webcited by 60
  7. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  8. 2021Boosting Search Engines with Interactive Agentscited by 19
  9. 2019Interactive Machine Comprehension with Information Seeking Agentscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 85; 2023: 341; 2024: 491; 2025: 430; 2026: 374.

1,722 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
38 more
  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  13. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  14. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  15. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  16. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  17. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  18. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  19. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  20. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  21. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  22. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  23. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  24. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  25. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  26. 2023A Survey on Multimodal Large Language Modelscited by 898
  27. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  28. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  29. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  30. 2022A Generalist Agentcited by 802
  31. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  32. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  33. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  34. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  35. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  36. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  37. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  38. 2023A Comprehensive Overview of Large Language Modelscited by 639

50 most cited of 1,722 All 1,722 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data
  2. 2026Search-GRT: Guided Retrieval Training of Search Agents to Optimize for Complex Question Answering
  3. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  4. 2023Aligning a medium-size GPT model in English to a small closed domain in Spanish
  5. 2026Replaying pre-training data improves fine-tuning
  6. 2023Axiomatic Preference Modeling for Longform Question Answering
  7. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequences
  8. 2025Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms