mlsanityData to 7 Oct 2026

Alignment of Language Agents

arXiv 2103.14659 ↗First posted 26 Mar 2021cs.AI cs.LGcited by 167

Not on the map. Why?

Cited by 49 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For artificial intelligence to be beneficial to humans the behaviour of AI agents needs to be aligned with what humans want. In this paper we discuss some behavioural issues for language agents, arising from accidental misspecification by the system designer. We highlight some ways that misspecification can occur and discuss some behavioural issues that could arise from misspecification, including deceptive or manipulative language, and review some approaches for avoiding these issues.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Builds on 39 of its 92 references that are in the map. First posted on arXiv 26 Mar 2021.

Builds on

39 of 92 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  10. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  11. 2020Artificial Intelligence, Values and Alignmentcited by 443
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
27 more
  1. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  2. 2018AI safety via debatecited by 384
  3. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  4. 2016Cooperative Inverse Reinforcement Learningcited by 298
  5. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  7. 2017Inverse Reward Designcited by 246
  8. 2018Gender Bias in Neural Natural Language Processingcited by 210
  9. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  10. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  11. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  12. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  13. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  14. 2018Supervising strong learners by amplifying weak expertscited by 139
  15. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  16. 2017AI Safety Gridworldscited by 108
  17. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  18. 2016The Off-Switch Gamecited by 97
  19. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  20. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  21. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 70
  22. 2021Agent Incentives: A Causal Perspectivecited by 51
  23. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  24. 2018AGI Safety Literature Reviewcited by 28
  25. 2017Good and safe uses of AI Oraclescited by 13
  26. 2021How RL Agents Behave When Their Actions Are Modifiedcited by 10
  27. 2019Modeling AGI Safety Frameworks with Causal Influence Diagramscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 16; 2023: 56; 2024: 42; 2025: 32; 2026: 14.

167 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2022A Generalist Agentcited by 802
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  6. 2021Unsolved Problems in ML Safetycited by 291
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023Large Language Model Alignment: A Surveycited by 241
  9. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  10. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  11. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  13. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  14. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  15. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  16. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  17. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  18. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  19. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  20. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  21. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  22. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  23. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  24. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  25. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  26. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  27. 2023Eight Things to Know about Large Language Modelscited by 73
  28. 2023Alignment for Honestycited by 72
  29. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 61
  30. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  31. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  32. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  33. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  34. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  35. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47
  36. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  37. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  38. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44

50 most cited of 167 All 167 (JSON)