mlsanityData to 7 Oct 2026

Improving alignment of dialogue agents via targeted human judgements

arXiv 2209.14375 ↗First posted 28 Sep 2022cs.LG cs.CLcited by 527

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.

Key facts. Cited by 527 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 69 of its 106 references that are in the map. First posted on arXiv 28 Sep 2022.

Builds on

69 of 106 references.

References · 69

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  13. 2017Selective Classification for Deep Neural Networkscited by 799
  14. 2020Aligning AI With Shared Human Valuescited by 706
  15. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  16. 2019ELI5: Long Form Question Answeringcited by 567
  17. 2018Gender Bias in Coreference Resolutioncited by 438
  18. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  19. 2018AI safety via debatecited by 384
  20. 2022Self-critiquing models for assisting human evaluatorscited by 372
  21. 2015Deep Learning with Limited Numerical Precisioncited by 368
  22. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  23. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  24. 2016Cooperative Inverse Reinforcement Learningcited by 298
  25. 2022Teaching language models to support answers with verified quotescited by 290
  26. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  27. 2020Selective Question Answering under Domain Shiftcited by 228
  28. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  29. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  30. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  31. 2021Challenges in Detoxifying Language Modelscited by 169
  32. 2018Supervising strong learners by amplifying weak expertscited by 139
  33. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  34. 2022Power to the People? Opportunities and Challenges for Participatory AIcited by 137
  35. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  36. 2020Explainable Deep Learning: A Field Guide for the Uninitiatedcited by 127
  37. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  38. 2020Participation is not a Design Fix for Machine Learningcited by 110

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 197; 2024: 176; 2025: 100; 2026: 39.

527 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
38 more
  1. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  5. 2023Whose Opinions Do Language Models Reflect?cited by 759
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  13. 2023A Comprehensive Overview of Large Language Modelscited by 639
  14. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  15. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  16. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  17. 2024Large Language Models: A Surveycited by 506
  18. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  19. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  20. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  21. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  22. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  23. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  24. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  25. 2023Detecting Language Model Attacks with Perplexitycited by 411
  26. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  27. 2023Augmented Language Models: a Surveycited by 398
  28. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  29. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  30. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  31. 2023Language Models can Solve Computer Taskscited by 371
  32. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  33. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  34. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  35. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  36. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  37. 2023Role-Play with Large Language Modelscited by 290
  38. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276

50 most cited of 527 All 527 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  2. 2024Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations
  3. 2026Learning to Simulate Human Dialogue
  4. 2023Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generation
  5. 2025Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition
  6. 2026Benevolent Bias in Multi-Turn Human-Agent Dialogue
  7. 2025InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context
  8. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback