# Let's Do a Thought Experiment: Using Counterfactuals to Improve Moral Reasoning

arXiv: https://arxiv.org/abs/2306.14308 · HTML: https://mlsanity.com/paper/2306.14308/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 4 of its 25 references that are in the map.
- First posted on arXiv 25 Jun 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 4 of 4)

- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [A Word on Machine Ethics: A Response to Jiang et al. (2021)](https://mlsanity.com/paper/2111.04158/) (2021)

## Built on by (most cited first, 20 of 20)

- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation](https://mlsanity.com/paper/2310.01320/) (2023)
- [CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving](https://mlsanity.com/paper/2404.17729/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses](https://mlsanity.com/paper/2312.00763/) (2023)
- [Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agents](https://mlsanity.com/paper/2501.00430/) (2024)
- [Improving Interpersonal Communication by Simulating Audiences with Language Models](https://mlsanity.com/paper/2311.00687/) (2023)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs](https://mlsanity.com/paper/2405.12933/) (2024)
- [Backtracking for Safety](https://mlsanity.com/paper/2503.08919/) (2025)
- [ProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMs](https://mlsanity.com/paper/2602.13274/) (2026)
- [SaGE: Evaluating Moral Consistency in Large Language Models](https://mlsanity.com/paper/2402.13709/) (2024)
- [Test-Time Fairness and Robustness in Large Language Models](https://mlsanity.com/paper/2406.07685/) (2024)
- [MTMT: Consolidating Multiple Thinking Modes to Form a Thought Tree for Strengthening LLM](https://mlsanity.com/paper/2412.03987/) (2024)
- [Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation](https://mlsanity.com/paper/2507.09839/) (2025)
- [Explainable Ethical Assessment on Human Behaviors by Generating Conflicting Social Norms](https://mlsanity.com/paper/2512.15793/) (2025)
- [Dynamic Analysis and Adaptive Discriminator for Fake News Detection](https://mlsanity.com/paper/2408.10883/) (2024)
- [ClarityEthic: Explainable Moral Judgment Utilizing Contrastive Ethical Insights from Large Language Models](https://mlsanity.com/paper/2412.12848/) (2024)
- [Reinforcement Learning with Backtracking Feedback](https://mlsanity.com/paper/2602.08377/) (2026)
- [Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs](https://mlsanity.com/paper/2606.12731/) (2026)

## Abstract (arXiv)

Language models still struggle on moral reasoning, despite their impressive performance in many other tasks. In particular, the Moral Scenarios task in MMLU (Multi-task Language Understanding) is among the worst performing tasks for many language models, including GPT-3. In this work, we propose a new prompting framework, Thought Experiments, to teach language models to do better moral reasoning using counterfactuals. Experiment results show that our framework elicits counterfactual questions and answers from the model, which in turn helps improve the accuracy on Moral Scenarios task by 9-16% compared to other zero-shot baselines. Interestingly, unlike math reasoning tasks, zero-shot Chain-of-Thought (CoT) reasoning doesn't work out of the box, and even reduces accuracy by around 4% compared to direct zero-shot. We further observed that with minimal human supervision in the form of 5 few-shot examples, the accuracy of the task can be improved to as much as 80%.