# Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theory

arXiv: https://arxiv.org/abs/2306.11432 · HTML: https://mlsanity.com/paper/2306.11432/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences](https://mlsanity.com/paper/2012.15738/) (2020)
- [UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark](https://mlsanity.com/paper/2103.13009/) (2021)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [Towards Answering Open-ended Ethical Quandary Questions](https://mlsanity.com/paper/2205.05989/) (2022)

## Built on by (most cited first, 8 of 8)

- [The Greatest Good Benchmark: Measuring LLMs' Alignment with Utilitarian Moral Dilemmas](https://mlsanity.com/paper/2503.19598/) (2025)
- [An Evaluation of GPT-4 on the ETHICS Dataset](https://mlsanity.com/paper/2309.10492/) (2023)
- [Histoires Morales: A French Dataset for Assessing Moral Alignment](https://mlsanity.com/paper/2501.17117/) (2025)
- [Intuitions of Compromise: Utilitarianism vs. Contractualism](https://mlsanity.com/paper/2410.05496/) (2024)
- [Explainable Ethical Assessment on Human Behaviors by Generating Conflicting Social Norms](https://mlsanity.com/paper/2512.15793/) (2025)
- [ClarityEthic: Explainable Moral Judgment Utilizing Contrastive Ethical Insights from Large Language Models](https://mlsanity.com/paper/2412.12848/) (2024)
- [Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models](https://mlsanity.com/paper/2506.14625/) (2025)
- [Bounded Morality: Defining the Space of Moral Computation](https://mlsanity.com/paper/2607.00002/) (2026)

## Abstract (arXiv)

Moral AI has been studied in the fields of philosophy and artificial intelligence. Although most existing studies are only theoretical, recent developments in AI have made it increasingly necessary to implement AI with morality. On the other hand, humans are under the moral uncertainty of not knowing what is morally right. In this paper, we implement the Maximizing Expected Choiceworthiness (MEC) algorithm, which aggregates outputs of models based on three normative theories of normative ethics to generate the most appropriate output. MEC is a method for making appropriate moral judgments under moral uncertainty. Our experimental results suggest that the output of MEC correlates to some extent with commonsense morality and that MEC can produce equally or more appropriate output than existing methods.