mlsanityData to 7 Oct 2026

Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?

arXiv 2308.15399 ↗First posted 29 Aug 2023cs.CLcited by 36

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Making moral judgments is an essential step toward developing ethical AI systems. Prevalent approaches are mostly implemented in a bottom-up manner, which uses a large set of annotated data to train models based on crowd-sourced opinions about morality. These approaches have been criticized for overgeneralizing the moral stances of a limited group of annotators and lacking explainability. This work proposes a flexible top-down framework to steer (Large) Language Models (LMs) to perform moral reasoning with well-established moral theories from interdisciplinary research. The theory-guided top-down framework can incorporate various moral theories. Our experiments demonstrate the effectiveness of the proposed framework on datasets derived from moral theories. Furthermore, we show the alignment between different moral theories and existing morality datasets. Our analysis exhibits the potential and flaws in existing resources (models and datasets) in developing explainable moral judgment-making systems.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 72 references that are in the map. First posted on arXiv 29 Aug 2023.

Builds on

36 of 72 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2020Aligning AI With Shared Human Valuescited by 706
  12. 2022Large Language Models Can Self-Improvecited by 656
24 more
  1. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  2. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  3. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  4. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  5. 2022Data Contamination: From Memorization to Exploitationcited by 234
  6. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  7. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  8. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  9. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
  10. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 100
  11. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  12. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 92
  13. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  14. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
  15. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 71
  16. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  17. 2023Knowledge of cultural moral norms in large language modelscited by 51
  18. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  19. 2021SWSR: A Chinese Dataset and Lexicon for Online Sexism Detectioncited by 42
  20. 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
  21. 2022ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situationscited by 36
  22. 2021ValueNet: A New Dataset for Human Value Driven Dialogue Systemcited by 34
  23. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  24. 2016The Dark Side of Ethical Robotscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 14; 2026: 7.

36 papers.

Sort:

By area:

  1. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  2. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 79
  3. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 52
  4. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 45
  5. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  6. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychologycited by 25
  7. 2023Probing the Moral Development of Large Language Models through Defining Issues Testcited by 23
  8. 2024MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussionscited by 23
  9. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  10. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  11. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  12. 2024Evaluating Gender Bias of LLMs in Making Morality Judgementscited by 15
24 more
  1. 2025When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmascited by 14
  2. 2025The Staircase of Ethics: Probing LLM Value Priorities through Multi-Step Induction to Complex Moral Dilemmascited by 13
  3. 2024Do Moral Judgment and Reasoning Capability of LLMs Change with Language? A Study using the Multilingual Defining Issues Testcited by 9
  4. 2025MFTCXplain: A Multilingual Benchmark Dataset for Evaluating the Moral Reasoning of LLMs through Multi-hop Hate Speech Explanationcited by 7
  5. 2025MoVa: Towards Generalizable Classification of Human Morals and Valuescited by 7
  6. 2025Auditing the Ethical Logic of Generative AI Modelscited by 5
  7. 2024Purple-teaming LLMs with Adversarial Defender Trainingcited by 4
  8. 2025One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoningcited by 4
  9. 2025RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivitycited by 4
  10. 2024Foundation Models for Autonomous Robots in Unstructured Environmentscited by 3
  11. 2025Moral Reasoning Across Languages: The Critical Role of Low-Resource Languages in LLMscited by 3
  12. 2025Deontological Keyword Bias: The Impact of Modal Expressions on Normative Judgments of Language Modelscited by 2
  13. 2025A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMscited by 2
  14. 2025Diverse Human Value Alignment for Large Language Models via Ethical Reasoningcited by 2
  15. 2026If LLMs Have Human-Like Attributes, Then So Does Age of Empires IIcited by 2
  16. 2024Speciesism in Natural Language Processing Researchcited by 1
  17. 2025Contemplative Artificial Intelligencecited by 1
  18. 2026Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainabilitycited by 1
  19. 2026Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Modelscited by 1
  20. 2025Normative Moral Pluralism for AI: A Framework for Deliberation in Complex Moral Contextscited by 0
  21. 2026Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domainscited by 0
  22. 2026An Algebraic Exposition of the Theory of Dyadic Moralitycited by 0
  23. 2026Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picturecited by 0
  24. 2026Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MoralBench: Moral Evaluation of LLMs
  2. 2026Are LLMs Bad at Moral Reasoning?
  3. 2026Beyond Binary Moral Judgment: Modeling Ethical Pluralism in AI
  4. 2025The Convergent Ethics of AI? Analyzing Moral Foundation Priorities in Large Language Models with a Multi-Framework Approach
  5. 2026Unsupervised Elicitation of Moral Values from Language Models
  6. 2025Are Language Models Consequentialist or Deontological Moral Reasoners?
  7. 2025Beyond Ethical Alignment: Evaluating LLMs as Artificial Moral Assistants
  8. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework