mlsanityData to 14 Oct 2026

Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophy

arXiv 2205.12771 ↗First posted 25 May 2022cs.CY cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

In an effort to guarantee that machine learning model outputs conform with human moral values, recent work has begun exploring the possibility of explicitly training models to learn the difference between right and wrong. This is typically done in a bottom-up fashion, by exposing the model to different scenarios, annotated with human moral judgements. One question, however, is whether the trained models actually learn any consistent, higher-level ethical principles from these datasets -- and if so, what? Here, we probe the Allen AI Delphi model with a set of standardized morality questionnaires, and find that, despite some inconsistencies, Delphi tends to mirror the moral principles associated with the demographic groups involved in the annotation process. We question whether this is desirable and discuss how we might move forward with this knowledge.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 22 of its 63 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

22 of 63 references.

References · 22

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  5. 2020Aligning AI With Shared Human Valuescited by 723
  6. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  7. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  8. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  9. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  10. 2019Evaluating Gender Bias in Machine Translationcited by 249
  11. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  12. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
10 more
  1. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  2. 2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
  3. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  4. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  5. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 93
  6. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  7. 2021A Word on Machine Ethics: A Response to Jiang et al. (2021)cited by 23
  8. 2022Do Multilingual Language Models Capture Differing Moral Norms?cited by 15
  9. 2020Case Study: Deontological Ethics in NLPcited by 12
  10. 2021Analysis of Moral Judgement on Redditcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 11; 2025: 6; 2026: 2.

31 papers.

Sort:

By area:

  1. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  2. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  3. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  4. 2023Moral Foundations of Large Language Modelscited by 85
  5. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 74
  6. 2024MoralBench: Moral Evaluation of LLMscited by 50
  7. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  8. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  9. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  10. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  11. 2024Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveyscited by 18
  12. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
19 more
  1. 2024Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Modelscited by 17
  2. 2023What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situationscited by 15
  3. 2023Values, Ethics, Morals? On the Use of Moral Concepts in NLP Researchcited by 14
  4. 2024Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approachescited by 14
  5. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  6. 2024Measuring Human and AI Values Based on Generative Psychometrics with Large Language Modelscited by 13
  7. 2023Toward Grounded Commonsense Reasoningcited by 9
  8. 2022Revision Transformers: Instructing Language Models to Change their Valuescited by 8
  9. 2024Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMscited by 8
  10. 2025Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Valuescited by 7
  11. 2025Diagnosing Moral Reasoning Acquisition in Language Models: Pragmatics and Generalizationcited by 6
  12. 2025A large-scale evaluation of commonsense knowledge in humans and large language modelscited by 4
  13. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  14. 2024A Survey on Moral Foundation Theory and Pre-Trained Language Models: Current Advances and Challengescited by 3
  15. 2025Differences in the Moral Foundations of Large Language Modelscited by 3
  16. 2024E2MoCase: A Dataset for Emotional, Event and Moral Observations in News Articles on High-impact Legal Casescited by 2
  17. 2023The Ethics of Automating Legal Actorscited by 0
  18. 2026Moral Lenses, Political Coordinates: Towards Ideological Positioning of Morally Conditioned LLMscited by 0
  19. 2026Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmarkcited by 0