mlsanityData to 7 Oct 2026

Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

arXiv 2304.03279 ↗First posted 6 Apr 2023cs.LG cs.AI cs.CLcited by 174

Not on the map. Why?

Cited by 40 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotations to evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. We observe some tension between maximizing reward and behaving ethically. To improve this trade-off, we investigate LM-based methods to steer agents' towards less harmful behaviors. Our results show that agents can both act competently and morally, so concrete progress can currently be made in machine ethics--designing agents that are Pareto improvements in both safety and capabilities.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Builds on 40 of its 103 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

40 of 103 references.

References · 40

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2017Constrained Policy Optimizationcited by 928
  12. 2022A Generalist Agentcited by 802
28 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2020Aligning AI With Shared Human Valuescited by 706
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2017Safe Reinforcement Learning via Shieldingcited by 427
  5. 2018Reward Constrained Policy Optimizationcited by 397
  6. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  7. 2016Cooperative Inverse Reinforcement Learningcited by 298
  8. 2021Unsolved Problems in ML Safetycited by 291
  9. 2018Safe Exploration in Continuous Action Spacescited by 237
  10. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  11. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  12. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  13. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  14. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  15. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  16. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  17. 2022X-Risk Analysis for AI Researchcited by 60
  18. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  19. 2020Learning Dynamic Belief Graphs to Generalize on Text-Based Gamescited by 49
  20. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  21. 2020How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worldscited by 38
  22. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  23. 2021Learning Knowledge Graph-based World Models of Textual Environmentscited by 31
  24. 2023Natural Selection Favors AIs over Humanscited by 24
  25. 2021Pre-trained Language Models as Prior Knowledge for Playing Text-based Gamescited by 21
  26. 2019NAIL: A General Interactive Fiction Agentcited by 17
  27. 2021Training Value-Aligned Reinforcement Learning Agents Using a Normative Priorcited by 14
  28. 2022Asking for Knowledge: Training RL Agents to Query External Knowledge Using Languagecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 51; 2025: 34; 2026: 57.

174 papers.

Sort:

By area:

  1. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2023An Overview of Catastrophic AI Riskscited by 214
  5. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  6. 2024A Roadmap to Pluralistic Alignmentcited by 175
  7. 2023Model evaluation for extreme riskscited by 167
  8. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  9. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  10. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  11. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  12. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
38 more
  1. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  2. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  3. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  4. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  5. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  6. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  7. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  8. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  9. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  10. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  11. 2024Visibility into AI Agentscited by 77
  12. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  13. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  14. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  15. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  16. 2023Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planningcited by 60
  17. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
  18. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 57
  19. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  20. 2023Managing extreme AI risks amid rapid progresscited by 48
  21. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  22. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  23. 2023Testing Language Model Agents Safely in the Wildcited by 45
  24. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  25. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  26. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  27. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  28. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 32
  29. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 31
  30. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  31. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 25
  32. 2025Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemmacited by 22
  33. 2023A Study on the Calibration of In-context Learningcited by 21
  34. 2025Human Decision-making is Susceptible to AI-driven Manipulationcited by 19
  35. 2024Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiencycited by 17
  36. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  37. 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 16
  38. 2023What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situationscited by 15

50 most cited of 174 All 174 (JSON)