# MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks

arXiv: https://arxiv.org/abs/2310.19677 · HTML: https://mlsanity.com/paper/2310.19677/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 41 of its 120 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [Revisiting the "Video" in Video-Language Understanding](https://mlsanity.com/paper/2206.01720/) (2022)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences](https://mlsanity.com/paper/2012.15738/) (2020)

## Built on by (most cited first, 30 of 44)

- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Aligning to Thousands of Preferences via System Message Generalization](https://mlsanity.com/paper/2405.17977/) (2024)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [Large Language Models and Causal Inference in Collaboration: A Survey](https://mlsanity.com/paper/2403.09606/) (2024)
- [Causality for Large Language Models](https://mlsanity.com/paper/2410.15319/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models](https://mlsanity.com/paper/2405.13974/) (2024)
- [Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domain](https://mlsanity.com/paper/2406.06435/) (2024)
- [CLEAR: Can Language Models Really Understand Causal Graphs?](https://mlsanity.com/paper/2406.16605/) (2024)
- [Counterfactual Token Generation in Large Language Models](https://mlsanity.com/paper/2409.17027/) (2024)
- [Understanding How Value Neurons Shape the Generation of Specified Values in LLMs](https://mlsanity.com/paper/2505.17712/) (2025)
- [When Do Large Language Models Exhibit Unsolicited Deception?](https://mlsanity.com/paper/2504.00285/) (2025)
- [CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives](https://mlsanity.com/paper/2504.10823/) (2025)
- [J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization](https://mlsanity.com/paper/2505.13346/) (2025)
- [Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios](https://mlsanity.com/paper/2407.19760/) (2024)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [Eagle: Ethical Dataset Given from Real Interactions](https://mlsanity.com/paper/2402.14258/) (2024)
- [Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Models](https://mlsanity.com/paper/2404.10975/) (2024)
- [SocialGaze: Improving the Integration of Human Social Norms in Large Language Models](https://mlsanity.com/paper/2410.08698/) (2024)
- [AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs](https://mlsanity.com/paper/2510.13912/) (2025)
- [Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection](https://mlsanity.com/paper/2601.09692/) (2026)
- [On the Context Sensitivity of LLM Moral Judgment](https://mlsanity.com/paper/2603.23114/) (2026)
- [When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysis](https://mlsanity.com/paper/2606.23196/) (2026)
- [STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions](https://mlsanity.com/paper/2409.13843/) (2024)
- [Large Language Models for Zero-shot Inference of Causal Structures in Biology](https://mlsanity.com/paper/2503.04347/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Are UFOs Driving Innovation? The Illusion of Causality in Large Language Models](https://mlsanity.com/paper/2410.11684/) (2024)
- [AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation](https://mlsanity.com/paper/2505.11887/) (2025)

## Abstract (arXiv)

Human commonsense understanding of the physical and social world is organized around intuitive theories. These theories support making causal and moral judgments. When something bad happens, we naturally ask: who did what, and why? A rich literature in cognitive science has studied people's causal and moral intuitions. This work has revealed a number of factors that systematically influence people's judgments, such as the violation of norms and whether the harm is avoidable or inevitable. We collected a dataset of stories from 24 cognitive science papers and developed a system to annotate each story with the factors they investigated. Using this dataset, we test whether large language models (LLMs) make causal and moral judgments about text-based scenarios that align with those of human participants. On the aggregate level, alignment has improved with more recent LLMs. However, using statistical analyses, we find that LLMs weigh the different factors quite differently from human participants. These results show how curated, challenge datasets combined with insights from cognitive science can help us go beyond comparisons based merely on aggregate metrics: we uncover LLMs implicit tendencies and show to what extent these align with human intuitions.