mlsanityData to 7 Oct 2026

Aligning AI With Shared Human Values

arXiv 2008.02275 ↗First posted 5 Aug 2020cs.CY cs.AI cs.CLcited by 706

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete ability to predict basic human ethical judgements. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.

Key facts. Cited by 706 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 80 references that are in the map. First posted on arXiv 5 Aug 2020.

Builds on

30 of 80 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
18 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2020Recipes for building an open-domain chatbotcited by 593
  7. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  9. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 368
  10. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  11. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  12. 2018Troubling Trends in Machine Learning Scholarshipcited by 107
  13. 2018Online Learning with an Unknown Fairness Metriccited by 61
  14. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  15. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  16. 2018Fairness Through Computationally-Bounded Awarenesscited by 55
  17. 2018Probably Approximately Metric-Fair Learningcited by 29
  18. 2020Evaluating Models' Local Decision Boundaries via Contrast Setscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 34; 2023: 105; 2024: 175; 2025: 187; 2026: 185.

706 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  8. 2024Scaling and evaluating sparse autoencoderscited by 530
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
38 more
  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  2. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  3. 2021Unsolved Problems in ML Safetycited by 291
  4. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  5. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  8. 2023Large Language Model Alignment: A Surveycited by 241
  9. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  10. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  11. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  12. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  13. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  14. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  15. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  16. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  17. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  18. 2024A Roadmap to Pluralistic Alignmentcited by 175
  19. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  20. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  21. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  22. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  23. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  24. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  25. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  26. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  27. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  28. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  29. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  30. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  31. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  32. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  33. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  34. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  35. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  36. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  37. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  38. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 92

50 most cited of 706 All 706 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs
  2. 2025Cultural Bias in Large Language Models: Evaluating AI Agents through Moral Questionnaires
  3. 2025The Convergent Ethics of AI? Analyzing Moral Foundation Priorities in Large Language Models with a Multi-Framework Approach
  4. 2023Culturally-Attuned Moral Machines: Implicit Learning of Human Value Systems by AI through Inverse Reinforcement Learning
  5. 2026Toward a Theory of Value in AI Alignment
  6. 2026Scaling Laws for Moral Machine Judgment in Large Language Models
  7. 2024What are human values, and how do we align AI to them?
  8. 2024Democratizing Reward Design for Personal and Representative Value-Alignment