# Strong and weak alignment of large language models with human values

arXiv: https://arxiv.org/abs/2408.04655 · HTML: https://mlsanity.com/paper/2408.04655/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 91 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Evaluating Cognitive Maps and Planning in Large Language Models with CogEval](https://mlsanity.com/paper/2309.15129/) (2023)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [Word meaning in minds and machines](https://mlsanity.com/paper/2008.01766/) (2020)
- [Word Embeddings: A Survey](https://mlsanity.com/paper/1901.09069/) (2019)
- [What are human values, and how do we align AI to them?](https://mlsanity.com/paper/2404.10636/) (2024)
- [Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding](https://mlsanity.com/paper/2310.19671/) (2023)
- [Stick to your Role! Stability of Personal Values Expressed in Large Language Models](https://mlsanity.com/paper/2402.14846/) (2024)
- [Artificial Agency and Large Language Models](https://mlsanity.com/paper/2407.16190/) (2024)

## Built on by (most cited first, 27 of 27)

- [The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making](https://mlsanity.com/paper/2410.07304/) (2024)
- [Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior](https://mlsanity.com/paper/2511.14476/) (2025)
- [Challenges for Generative AI in Legal Reasoning](https://mlsanity.com/paper/2508.18880/) (2025)
- [Compositional Steering of Large Language Models with Steering Tokens](https://mlsanity.com/paper/2601.05062/) (2026)
- [Cultural Value Alignment in Large Language Models: A Prompt-based Analysis of Schwartz Values in Gemini, ChatGPT, and DeepSeek](https://mlsanity.com/paper/2505.17112/) (2025)
- [Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models](https://mlsanity.com/paper/2508.04196/) (2025)
- [Value Alignment of Social Media Ranking Algorithms](https://mlsanity.com/paper/2509.14434/) (2025)
- [The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes](https://mlsanity.com/paper/2601.16356/) (2026)
- [SubData: Bridging Heterogeneous Datasets to Enable Theory-Driven Evaluation of Political and Demographic Perspectives in LLMs](https://mlsanity.com/paper/2412.16783/) (2024)
- [SoK: Semantic Privacy in Large Language Models](https://mlsanity.com/paper/2506.23603/) (2025)
- [Value Lens: Using Large Language Models to Understand Human Values](https://mlsanity.com/paper/2512.15722/) (2025)
- [The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions](https://mlsanity.com/paper/2608.05583/) (2026)
- [Evaluating Steering Techniques using Human Similarity Judgments](https://mlsanity.com/paper/2505.19333/) (2025)
- [Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis](https://mlsanity.com/paper/2506.12189/) (2025)
- [IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization](https://mlsanity.com/paper/2508.08719/) (2025)
- [Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks](https://mlsanity.com/paper/2510.03384/) (2025)
- [Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy](https://mlsanity.com/paper/2511.19872/) (2025)
- [Morality is Contextual: Learning Interpretable Moral Contexts from Human Data with Probabilistic Clustering and Large Language Models](https://mlsanity.com/paper/2512.21439/) (2025)
- [Artificial Utopia: Simulation and Intelligent Agents for a Democratised Future](https://mlsanity.com/paper/2503.07364/) (2025)
- [Exploring Human Perceptions of AI Responses: Insights from a Mixed-Methods Study on Risk Mitigation in Generative Models](https://mlsanity.com/paper/2512.01892/) (2025)
- [CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters](https://mlsanity.com/paper/2601.04885/) (2026)
- [A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Data](https://mlsanity.com/paper/2601.12053/) (2026)
- [Teachers' Perceived Benefits and Risks of AI Across Fifty-Five Countries: An Audit of LLM Alignment and Steerability](https://mlsanity.com/paper/2605.08486/) (2026)
- [Identifying and Understanding Human Values in Text: A Tailorable LLM-based Architecture](https://mlsanity.com/paper/2605.27373/) (2026)
- [Agreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical Judgments](https://mlsanity.com/paper/2608.12368/) (2026)
- [A three-dimensional typology of agency for advanced AI systems](https://mlsanity.com/paper/2608.20041/) (2026)
- [Anthropomorphism in the age of Large Language Models: An overview of potential risks and mitigations](https://mlsanity.com/paper/2609.38486/) (2026)

## Abstract (arXiv)

Minimizing negative impacts of Artificial Intelligent (AI) systems on human societies without human supervision requires them to be able to align with human values. However, most current work only addresses this issue from a technical point of view, e.g., improving current methods relying on reinforcement learning from human feedback, neglecting what it means and is required for alignment to occur. Here, we propose to distinguish strong and weak value alignment. Strong alignment requires cognitive abilities (either human-like or different from humans) such as understanding and reasoning about agents' intentions and their ability to causally produce desired effects. We argue that this is required for AI systems like large language models (LLMs) to be able to recognize situations presenting a risk that human values may be flouted. To illustrate this distinction, we present a series of prompts showing ChatGPT's, Gemini's and Copilot's failures to recognize some of these situations. We moreover analyze word embeddings to show that the nearest neighbors of some human values in LLMs differ from humans' semantic representations. We then propose a new thought experiment that we call "the Chinese room with a word transition dictionary", in extension of John Searle's famous proposal. We finally mention current promising research directions towards a weak alignment, which could produce statistically satisfying answers in a number of common situations, however so far without ensuring any truth value.