# Warmth and competence in human-agent cooperation

arXiv: https://arxiv.org/abs/2201.13448 · HTML: https://mlsanity.com/paper/2201.13448/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 112 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Environmental drivers of systematicity and generalization in a situated agent](https://mlsanity.com/paper/1910.00571/) (2019)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [Maintaining cooperation in complex social dilemmas using deep reinforcement learning](https://mlsanity.com/paper/1707.01068/) (2017)
- [Conceptual Metaphors Impact Perceptions of Human-AI Collaboration](https://mlsanity.com/paper/2008.02311/) (2020)
- [Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi](https://mlsanity.com/paper/2107.07630/) (2021)
- [The Challenge of Value Alignment: from Fairer Algorithms to AI Safety](https://mlsanity.com/paper/2101.06060/) (2021)
- [AvE: Assistance via Empowerment](https://mlsanity.com/paper/2006.14796/) (2020)
- [Rotation, Translation, and Cropping for Zero-Shot Generalization](https://mlsanity.com/paper/2001.09908/) (2020)
- [Consequentialist conditional cooperation in social dilemmas with imperfect information](https://mlsanity.com/paper/1710.06975/) (2017)
- [Long-Term Trends in the Public Perception of Artificial Intelligence](https://mlsanity.com/paper/1609.04904/) (2016)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Exciting, Useful, Worrying, Futuristic: Public Perception of Artificial Intelligence in 8 Countries](https://mlsanity.com/paper/2001.00081/) (2019)
- [D3C: Reducing the Price of Anarchy in Multi-Agent Learning](https://mlsanity.com/paper/2010.00575/) (2020)
- [DeepMind Lab2D](https://mlsanity.com/paper/2011.07027/) (2020)
- [Levels of Analysis for Machine Learning](https://mlsanity.com/paper/2004.05107/) (2020)
- [Human-Agent Cooperation in Bridge Bidding](https://mlsanity.com/paper/2011.14124/) (2020)
- [Warmth and Competence to Predict Human Preference of Robot Behavior in Physical Human-Robot Interaction](https://mlsanity.com/paper/2008.05799/) (2020)

## Built on by (most cited first, 11 of 11)

- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Can Large Language Model Agents Simulate Human Trust Behavior?](https://mlsanity.com/paper/2402.04559/) (2024)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance](https://mlsanity.com/paper/2407.07950/) (2024)
- [Measuring and mitigating overreliance to build human-compatible AI](https://mlsanity.com/paper/2509.08010/) (2025)
- [Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspective](https://mlsanity.com/paper/2304.11632/) (2023)
- [Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmas](https://mlsanity.com/paper/2305.00768/) (2023)
- [Human-AI Collaboration: Trade-offs Between Performance and Preferences](https://mlsanity.com/paper/2503.00248/) (2025)
- [Recourse for reclamation: Chatting with generative language models](https://mlsanity.com/paper/2403.14467/) (2024)
- [Moving Out: Physically-grounded Human-AI Collaboration](https://mlsanity.com/paper/2507.18623/) (2025)
- [Bounded Normative Equivalence in Human-AI Cooperation: Group Behaviour, Not Partner Labels, Predicts Cooperation under Anonymous Aggregate Feedback](https://mlsanity.com/paper/2601.20487/) (2026)

## Abstract (arXiv)

Interaction and cooperation with humans are overarching aspirations of artificial intelligence (AI) research. Recent studies demonstrate that AI agents trained with deep reinforcement learning are capable of collaborating with humans. These studies primarily evaluate human compatibility through "objective" metrics such as task performance, obscuring potential variation in the levels of trust and subjective preference that different agents garner. To better understand the factors shaping subjective preferences in human-agent cooperation, we train deep reinforcement learning agents in Coins, a two-player social dilemma. We recruit N = 501 participants for a human-agent cooperation study and measure their impressions of the agents they encounter. Participants' perceptions of warmth and competence predict their stated preferences for different agents, above and beyond objective performance metrics. Drawing inspiration from social science and biology research, we subsequently implement a new ``partner choice'' framework to elicit revealed preferences: after playing an episode with an agent, participants are asked whether they would like to play the next episode with the same agent or to play alone. As with stated preferences, social perception better predicts participants' revealed preferences than does objective performance. Given these results, we recommend human-agent interaction researchers routinely incorporate the measurement of social perception and subjective preferences into their studies.