mlsanityData to 14 Oct 2026

Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models

arXiv 2506.14625 ↗First posted 17 Jun 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive moral reasoning abilities. Yet they often diverge when confronted with complex, multi-factor moral dilemmas. To address these discrepancies, we propose a framework that synthesizes multiple LLMs' moral judgments into a collectively formulated moral judgment, realigning models that deviate significantly from this consensus. Our aggregation mechanism fuses continuous moral acceptability scores (beyond binary labels) into a collective probability, weighting contributions by model reliability. For misaligned models, a targeted embedding-optimization procedure fine-tunes token embeddings for moral philosophical theories, minimizing JS divergence to the consensus while preserving semantic integrity. Experiments on a large-scale social moral dilemma dataset show our approach builds robust consensus and improves individual model fidelity. These findings highlight the value of data-driven moral alignment across multiple models and its potential for safer, more consistent AI systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 39 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

14 of 39 references.

References · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2022Mass-Editing Memory in a Transformercited by 1,001
  10. 2021Fast Model Editing at Scalecited by 498
  11. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  12. 2024MoralBench: Moral Evaluation of LLMscited by 50
2 more
  1. 2022Do Multilingual Language Models Capture Differing Moral Norms?cited by 15
  2. 2023Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theorycited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mechanistic Origin of Moral Indifference in Language Models
  2. 2026Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models
  3. 2026Unsupervised Elicitation of Moral Values from Language Models
  4. 2024The Moral Mind(s) of Large Language Models
  5. 2026Representational alignment yields generalizable safety in language models
  6. 2026Beyond Binary Moral Judgment: Modeling Ethical Pluralism in AI
  7. 2026Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability
  8. 2025Diverse Human Value Alignment for Large Language Models via Ethical Reasoning