# Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making

arXiv: https://arxiv.org/abs/2503.05724 · HTML: https://mlsanity.com/paper/2503.05724/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 81 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [LLM Voting: Human Choices and AI Collective Decision Making](https://mlsanity.com/paper/2402.01766/) (2024)
- [FIRE: Fact-checking with Iterative Retrieval and Verification](https://mlsanity.com/paper/2411.00784/) (2024)
- [Navigating LLM Ethics: Advancements, Challenges, and Future Directions](https://mlsanity.com/paper/2406.18841/) (2024)
- [A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents](https://mlsanity.com/paper/1712.04172/) (2017)
- [Implementations in Machine Ethics: A Survey](https://mlsanity.com/paper/2001.07573/) (2020)
- [Reinforcement Learning Under Moral Uncertainty](https://mlsanity.com/paper/2006.04734/) (2020)
- [Ethics and Responsible AI Deployment](https://mlsanity.com/paper/2311.14705/) (2023)
- [A Formalization of Kant's Second Formulation of the Categorical Imperative](https://mlsanity.com/paper/1801.03160/) (2018)

## Built on by (most cited first, 3 of 3)

- [Cross-cultural value alignment frameworks for responsible AI governance: Evidence from China-West comparative analysis](https://mlsanity.com/paper/2511.17256/) (2025)
- [Moral Lenses, Political Coordinates: Towards Ideological Positioning of Morally Conditioned LLMs](https://mlsanity.com/paper/2601.08634/) (2026)
- [Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models](https://mlsanity.com/paper/2605.03609/) (2026)

## Abstract (arXiv)

We present an ethical decision-making framework that refines a pre-trained reinforcement learning (RL) model using a task-agnostic ethical layer. Following initial training, the RL model undergoes ethical fine-tuning, where human feedback is replaced by feedback generated from a large language model (LLM). The LLM embodies consequentialist, deontological, virtue, social justice, and care ethics as moral principles to assign belief values to recommended actions during ethical decision-making. An ethical layer aggregates belief scores from multiple LLM-derived moral perspectives using Belief Jensen-Shannon Divergence and Dempster-Shafer Theory into probability scores that also serve as the shaping reward, steering the agent toward choices that align with a balanced ethical framework. This integrated learning framework helps the RL agent navigate moral uncertainty in complex environments and enables it to make morally sound decisions across diverse tasks. Our approach, tested across different LLM variants and compared with other belief aggregation techniques, demonstrates improved consistency, adaptability, and reduced reliance on handcrafted ethical rewards. This method is especially effective in dynamic scenarios where ethical challenges arise unexpectedly, making it well-suited for real-world applications.