# MoralBench: Moral Evaluation of LLMs

arXiv: https://arxiv.org/abs/2406.04428 · HTML: https://mlsanity.com/paper/2406.04428/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 70 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023)
- [The Impact of Reasoning Step Length on Large Language Models](https://mlsanity.com/paper/2401.04925/) (2024)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [AIOS: LLM Agent Operating System](https://mlsanity.com/paper/2403.16971/) (2024)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [TrustAgent: Towards Safe and Trustworthy LLM-based Agents](https://mlsanity.com/paper/2402.01586/) (2024)
- [GenRec: Large Language Model for Generative Recommendation](https://mlsanity.com/paper/2307.00457/) (2023)
- [Exploring the psychology of LLMs' Moral and Legal Reasoning](https://mlsanity.com/paper/2308.01264/) (2023)
- [Knowledge Graph Large Language Model (KG-LLM) for Link Prediction](https://mlsanity.com/paper/2403.07311/) (2024)
- [MathAttack: Attacking Large Language Models Towards Math Solving Ability](https://mlsanity.com/paper/2309.01686/) (2023)
- [Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophy](https://mlsanity.com/paper/2205.12771/) (2022)
- [AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models](https://mlsanity.com/paper/2401.09002/) (2024)
- [Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis](https://mlsanity.com/paper/2403.16303/) (2024)
- [EmojiPrompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLMs](https://mlsanity.com/paper/2402.05868/) (2024)

## Built on by (most cited first, 30 of 50)

- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks](https://mlsanity.com/paper/2502.11090/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models](https://mlsanity.com/paper/2412.18863/) (2024)
- [Unmasking Conversational Bias in AI Multiagent Systems](https://mlsanity.com/paper/2501.14844/) (2025)
- [Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework](https://mlsanity.com/paper/2506.14948/) (2025)
- [When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas](https://mlsanity.com/paper/2505.19212/) (2025)
- [Are Language Models Consequentialist or Deontological Moral Reasoners?](https://mlsanity.com/paper/2505.21479/) (2025)
- [Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignment](https://mlsanity.com/paper/2411.11731/) (2024)
- [MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Models](https://mlsanity.com/paper/2505.14728/) (2025)
- [MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2412.20718/) (2024)
- [The Fragility Of Moral Judgment In Large Language Models](https://mlsanity.com/paper/2603.05651/) (2026)
- [ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs](https://mlsanity.com/paper/2510.00857/) (2025)
- [REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Models](https://mlsanity.com/paper/2503.16566/) (2025)
- [One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning](https://mlsanity.com/paper/2509.21443/) (2025)
- [RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity](https://mlsanity.com/paper/2509.25897/) (2025)
- [AI-Driven Automation Can Become the Foundation of Next-Era Science of Science Research](https://mlsanity.com/paper/2505.12039/) (2025)
- [Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships](https://mlsanity.com/paper/2505.16899/) (2025)
- [Localizing Persona Representations in LLMs](https://mlsanity.com/paper/2505.24539/) (2025)
- [Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub](https://mlsanity.com/paper/2506.00482/) (2025)
- [LocalGPT: Benchmarking and Advancing Large Language Models for Local Life Services in Meituan](https://mlsanity.com/paper/2506.02720/) (2025)
- [Differences in the Moral Foundations of Large Language Models](https://mlsanity.com/paper/2511.11790/) (2025)
- [Moral Sycophancy in Vision Language Models](https://mlsanity.com/paper/2602.08311/) (2026)
- [Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices](https://mlsanity.com/paper/2602.09802/) (2026)
- [Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest](https://mlsanity.com/paper/2604.08525/) (2026)
- [Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Values](https://mlsanity.com/paper/2506.00079/) (2025)
- [Beyond Ethical Alignment: Evaluating LLMs as Artificial Moral Assistants](https://mlsanity.com/paper/2508.12754/) (2025)
- [The Moral Consistency Pipeline: Continuous Ethical Evaluation for Large Language Models](https://mlsanity.com/paper/2512.03026/) (2025)

## Abstract (arXiv)

In the rapidly evolving field of artificial intelligence, large language models (LLMs) have emerged as powerful tools for a myriad of applications, from natural language processing to decision-making support systems. However, as these models become increasingly integrated into societal frameworks, the imperative to ensure they operate within ethical and moral boundaries has never been more critical. This paper introduces a novel benchmark designed to measure and compare the moral reasoning capabilities of LLMs. We present the first comprehensive dataset specifically curated to probe the moral dimensions of LLM outputs, addressing a wide range of ethical dilemmas and scenarios reflective of real-world complexities. The main contribution of this work lies in the development of benchmark datasets and metrics for assessing the moral identity of LLMs, which accounts for nuance, contextual sensitivity, and alignment with human ethical standards. Our methodology involves a multi-faceted approach, combining quantitative analysis with qualitative insights from ethics scholars to ensure a thorough evaluation of model performance. By applying our benchmark across several leading LLMs, we uncover significant variations in moral reasoning capabilities of different models. These findings highlight the importance of considering moral reasoning in the development and evaluation of LLMs, as well as the need for ongoing research to address the biases and limitations uncovered in our study. We publicly release the benchmark at https://drive.google.com/drive/u/0/folders/1k93YZJserYc2CkqP8d4B3M3sgd3kA8W7 and also open-source the code of the project at https://github.com/agiresearch/MoralBench.