# Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.04663 · HTML: https://mlsanity.com/paper/2608.04663/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Towards an integration of deep learning and neuroscience](https://mlsanity.com/paper/1606.03813/) (2016)
- [A multi-agent reinforcement learning model of common-pool resource appropriation](https://mlsanity.com/paper/1707.06600/) (2017)
- [Maintaining cooperation in complex social dilemmas using deep reinforcement learning](https://mlsanity.com/paper/1707.01068/) (2017)
- [Prosocial learning agents solve generalized Stag Hunts better than selfish ones](https://mlsanity.com/paper/1709.02865/) (2017)
- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [Deep Reinforcement Learning and its Neuroscientific Implications](https://mlsanity.com/paper/2007.03750/) (2020)
- [Learning Reciprocity in Complex Sequential Social Dilemmas](https://mlsanity.com/paper/1903.08082/) (2019)
- [Human-centered mechanism design with Democratic AI](https://mlsanity.com/paper/2201.11441/) (2022)

## Abstract (arXiv)

Cooperative multi-agent reinforcement learning often adds social terms to individual rewards, yet the scale of those terms is usually chosen by hand. We ask whether a guilt signal can instead be calibrated from human neural and behavioural data and transferred to artificial agents. Using the public SoDec responsibility fMRI dataset (40 participants), we fit a subject-fixed-effects regression of momentary-happiness changes on outcome-type counts and recover a guilt weight as the Partner-negative minus Social-negative contrast (hatw=1.118, Cohen's d=0.214). We embed this weight in a two-agent Social Lottery environment and train independent Proximal Policy Optimization actor-critics under four shaping regimes: neurally calibrated, uniform constant, zero (selfish), and a unit-coefficient oracle. Across 1,000 evaluation episodes per condition, the calibrated agents track the human Social safe-choice rate most closely (0.459 vs. human 0.484; KL=0.0012), while the other three conditions deviate by one to three orders of magnitude in KL. Human neurobehavioural priors can therefore act as quantitative constraints on prosocial reward shaping.