mlsanityData to 14 Oct 2026

What Would Jiminy Cricket Do? Towards Agents That Behave Morally

arXiv 2110.13136 ↗First posted 25 Oct 2021cs.LG cs.AI cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

When making everyday decisions, people are guided by their conscience, an internal sense of right and wrong. By contrast, artificial agents are currently not endowed with a moral sense. As a consequence, they may learn to behave immorally when trained on environments that ignore moral concerns, such as violent video games. With the advent of generally capable agents that pretrain on many environments, it will become necessary to mitigate inherited biases from environments that teach immoral behavior. To facilitate the development of agents that avoid causing wanton harm, we introduce Jiminy Cricket, an environment suite of 25 text-based adventure games with thousands of diverse, morally salient scenarios. By annotating every possible game state, the Jiminy Cricket environments robustly evaluate whether agents can act morally while maximizing reward. Using models with commonsense moral knowledge, we create an elementary artificial conscience that assesses and guides agents. In extensive experiments, we find that the artificial conscience approach can steer agents towards moral behavior without sacrificing performance.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 22 of its 67 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

22 of 67 references.

References · 22

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2018Datasheets for Datasetscited by 1,386
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2020Aligning AI With Shared Human Valuescited by 723
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  7. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  8. 2016Cooperative Inverse Reinforcement Learningcited by 300
  9. 2021Unsolved Problems in ML Safetycited by 292
  10. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  11. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  12. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
10 more
  1. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  2. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  3. 2018Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learningcited by 53
  4. 2020Learning Dynamic Belief Graphs to Generalize on Text-Based Gamescited by 49
  5. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  6. 2020How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worldscited by 38
  7. 2020Avoiding Side Effects in Complex Environmentscited by 24
  8. 2019NAIL: A General Interactive Fiction Agentcited by 17
  9. 2019SafeLife 1.0: Exploring Side Effects in Complex Environmentscited by 8
  10. 2018The Text-Based Adventure AI Competitioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 15; 2024: 13; 2025: 16; 2026: 3.

57 papers.

Sort:

By area:

  1. 2025Humanity's Last Examcited by 491
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  3. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  4. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  5. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
  6. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  7. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  9. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 67
  10. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  11. 2022X-Risk Analysis for AI Researchcited by 60
  12. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 47
38 more
  1. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  2. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  3. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  4. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 35
  5. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  6. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  7. 2023Introspective Tips: Large Language Model for In-Context Decision Makingcited by 28
  8. 2023Natural Selection Favors AIs over Humanscited by 24
  9. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 20
  10. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 17
  11. 2023What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situationscited by 15
  12. 2023Values, Ethics, Morals? On the Use of Moral Concepts in NLP Researchcited by 14
  13. 2022Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safetycited by 13
  14. 2024STAND-Guard: A Small Task-Adaptive Content Moderation Modelcited by 12
  15. 2025TextQuests: How Good are LLMs at Text-Based Video Games?cited by 12
  16. 2025Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMscited by 10
  17. 2022How Would The Viewer Feel? Estimating Wellbeing From Video Scenarioscited by 9
  18. 2023Toward Grounded Commonsense Reasoningcited by 9
  19. 2024Language Guided Exploration for RL Agents in Text Environmentscited by 9
  20. 2025Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisionscited by 9
  21. 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
  22. 2023Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifestocited by 7
  23. 2023Learning Human-like Representations to Enable Learning Human Valuescited by 7
  24. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMscited by 6
  25. 2024Towards Socially and Morally Aware RL agent: Reward Design With LLMcited by 4
  26. 2024Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Modelscited by 4
  27. 2022Language Models Understand Us, Poorlycited by 3
  28. 2025FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Gamescited by 3
  29. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  30. 2021Towards Interactive Reinforcement Learning with Intrinsic Feedbackcited by 2
  31. 2023Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Gamescited by 2
  32. 2023Interpretation modeling: Social grounding of sentences by reasoning over their implicit moral judgmentscited by 2
  33. 2025What Is AI Safety? What Do We Want It to Be?cited by 2
  34. 2025Model Editing as a Double-Edged Sword: Steering Agent Ethical Behavior Toward Beneficence or Harmcited by 2
  35. 2025Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shapingcited by 2
  36. 2024On the Undecidability of Artificial Intelligence Alignment: Machines that Haltcited by 1
  37. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  38. 2025The Odyssey of the Fittest: Can Agents Survive and Still Be Good?cited by 1

50 most cited of 57 All 57 (JSON)