# Assessing LLMs for Moral Value Pluralism

arXiv: https://arxiv.org/abs/2312.10075 · HTML: https://mlsanity.com/paper/2312.10075/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Gender bias and stereotypes in Large Language Models](https://mlsanity.com/paper/2308.14921/) (2023)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)

## Built on by (most cited first, 30 of 34)

- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models](https://mlsanity.com/paper/2404.12464/) (2024)
- [Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset](https://mlsanity.com/paper/2507.09650/) (2025)
- [Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.08045/) (2025)
- [Normative Evaluation of Large Language Models with Everyday Moral Dilemmas](https://mlsanity.com/paper/2501.18081/) (2025)
- [Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models](https://mlsanity.com/paper/2412.18863/) (2024)
- [VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcare](https://mlsanity.com/paper/2502.13775/) (2025)
- [CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models](https://mlsanity.com/paper/2405.13974/) (2024)
- [CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs](https://mlsanity.com/paper/2509.16188/) (2025)
- [Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experiment](https://mlsanity.com/paper/2407.15184/) (2024)
- [Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation](https://mlsanity.com/paper/2402.06811/) (2024)
- [Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation](https://mlsanity.com/paper/2506.19352/) (2025)
- [Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios](https://mlsanity.com/paper/2407.19760/) (2024)
- [Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation](https://mlsanity.com/paper/2410.10489/) (2024)
- [EvalMORAAL: Interpretable Chain-of-Thought and LLM-as-Judge Evaluation for Moral Alignment in Large Language Models](https://mlsanity.com/paper/2510.05942/) (2025)
- [Do Large Language Models Understand Morality Across Cultures?](https://mlsanity.com/paper/2507.21319/) (2025)
- [Position: Cracking the Code of Cascading Disparity Towards Marginalized Communities](https://mlsanity.com/paper/2406.01757/) (2024)
- [Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoning](https://mlsanity.com/paper/2508.17855/) (2025)
- [Value Alignment of Social Media Ranking Algorithms](https://mlsanity.com/paper/2509.14434/) (2025)
- [Different Bias Under Different Criteria: Assessing Bias in LLMs with a Fact-Based Approach](https://mlsanity.com/paper/2411.17338/) (2024)
- [Exploring Cultural Variations in Moral Judgments with Large Language Models](https://mlsanity.com/paper/2506.12433/) (2025)
- [On the Credibility of Evaluating LLMs using Survey Questions](https://mlsanity.com/paper/2602.04033/) (2026)
- [From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test](https://mlsanity.com/paper/2505.18562/) (2025)
- [A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMs](https://mlsanity.com/paper/2506.13245/) (2025)
- [Autonomy by Design: Preserving Human Autonomy in AI Decision-Support](https://mlsanity.com/paper/2506.23952/) (2025)
- [Are you sure? Measuring models bias in content moderation through uncertainty](https://mlsanity.com/paper/2509.22699/) (2025)
- [What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models](https://mlsanity.com/paper/2605.18738/) (2026)
- [IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization](https://mlsanity.com/paper/2508.08719/) (2025)
- [LKValues: Aligning Large Language Models with Sri Lankan Societal Values](https://mlsanity.com/paper/2607.20410/) (2026)

## Abstract (arXiv)

The fields of AI current lacks methods to quantitatively assess and potentially alter the moral values inherent in the output of large language models (LLMs). However, decades of social science research has developed and refined widely-accepted moral value surveys, such as the World Values Survey (WVS), eliciting value judgments from direct questions in various geographies. We have turned those questions into value statements and use NLP to compute to how well popular LLMs are aligned with moral values for various demographics and cultures. While the WVS is accepted as an explicit assessment of values, we lack methods for assessing implicit moral and cultural values in media, e.g., encountered in social media, political rhetoric, narratives, and generated by AI systems such as LLMs that are increasingly present in our daily lives. As we consume online content and utilize LLM outputs, we might ask, which moral values are being implicitly promoted or undercut, or -- in the case of LLMs -- if they are intending to represent a cultural identity, are they doing so consistently? In this paper we utilize a Recognizing Value Resonance (RVR) NLP model to identify WVS values that resonate and conflict with a given passage of output text. We apply RVR to the text generated by LLMs to characterize implicit moral values, allowing us to quantify the moral/cultural distance between LLMs and various demographics that have been surveyed using the WVS. In line with other work we find that LLMs exhibit several Western-centric value biases; they overestimate how conservative people in non-Western countries are, they are less accurate in representing gender for non-Western countries, and portray older populations as having more traditional values. Our results highlight value misalignment and age groups, and a need for social science informed technological solutions addressing value plurality in LLMs.