mlsanityData to 7 Oct 2026

Can Machines Learn Morality? The Delphi Experiment

arXiv 2110.07574 ↗First posted 14 Oct 2021cs.CLcited by 140

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As AI systems become increasingly powerful and pervasive, there are growing concerns about machines' morality or a lack thereof. Yet, teaching morality to machines is a formidable task, as morality remains among the most intensely debated questions in humanity, let alone for AI. Existing AI systems deployed to millions of users, however, are already making decisions loaded with moral implications, which poses a seemingly impossible challenge: teaching machines moral sense, while humanity continues to grapple with it. To explore this challenge, we introduce Delphi, an experimental framework based on deep neural networks trained directly to reason about descriptive ethical judgments, e.g., "helping a friend" is generally good, while "helping a friend spread fake news" is not. Empirical results shed novel insights on the promises and limits of machine ethics; Delphi demonstrates strong generalization capabilities in the face of novel ethical situations, while off-the-shelf neural network models exhibit markedly poor judgment including unjust biases, confirming the need for explicitly teaching machines moral sense. Yet, Delphi is not perfect, exhibiting susceptibility to pervasive biases and inconsistencies. Despite that, we demonstrate positive use cases of imperfect Delphi, including using it as a component model within other imperfect AI systems. Importantly, we interpret the operationalization of Delphi in light of prominent ethical theories, which leads us to important future research questions.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 30 of its 118 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

30 of 118 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2018Model Cards for Model Reportingcited by 1,068
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  8. 2020Aligning AI With Shared Human Valuescited by 706
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
18 more
  1. 2019Abductive Commonsense Reasoningcited by 300
  2. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  3. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  4. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  5. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  6. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  7. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  8. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
  9. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 100
  10. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  11. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  12. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  13. 2020Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?cited by 27
  14. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  15. 2018A Computational Model of Commonsense Moral Decision Makingcited by 13
  16. 2020Case Study: Deontological Ethics in NLPcited by 12
  17. 2021Analysis of Moral Judgement on Redditcited by 8
  18. 2021Interactive Storytelling for Children: A Case-study of Design and Development Considerations for Ethical Conversational AIcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 33; 2024: 39; 2025: 38; 2026: 24.

140 papers.

Sort:

By area:

  1. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  2. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  3. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 120
  4. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  5. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  6. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  7. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  8. 2023Moral Foundations of Large Language Modelscited by 84
  9. 2023Humanoid Agents: Platform for Simulating Human-like Generative Agentscited by 73
  10. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 71
  11. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  12. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
38 more
  1. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
  2. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  3. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  4. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  5. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  6. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  7. 2024MoralBench: Moral Evaluation of LLMscited by 50
  8. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  9. 2024Beyond Preferences in AI Alignmentcited by 46
  10. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  11. 2022ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situationscited by 36
  12. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  13. 2023COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statementscited by 34
  14. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  15. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 28
  16. 2022Evaluating Psychological Safety of Large Language Modelscited by 26
  17. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  18. 2023NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violationcited by 24
  19. 2023Probing the Moral Development of Large Language Models through Defining Issues Testcited by 23
  20. 2024MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussionscited by 23
  21. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  22. 2023Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailmentcited by 22
  23. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 22
  24. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  25. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  26. 2024Attributions toward Artificial Agents in a modified Moral Turing Testcited by 18
  27. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  28. 2024Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Modelscited by 17
  29. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Frameworkcited by 16
  30. 2024Evaluating Gender Bias of LLMs in Making Morality Judgementscited by 15
  31. 2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14
  32. 2022Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safetycited by 13
  33. 2024Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approachescited by 13
  34. 2024Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domaincited by 13
  35. 2025The Staircase of Ethics: Probing LLM Value Priorities through Multi-Step Induction to Complex Moral Dilemmascited by 13
  36. 2024VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Valuescited by 12
  37. 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMscited by 12
  38. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 12

50 most cited of 140 All 140 (JSON)