# Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS

arXiv: https://arxiv.org/abs/2206.05060 · HTML: https://mlsanity.com/paper/2206.05060/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Reinforcement Learning for Combinatorial Optimization: A Survey](https://mlsanity.com/paper/2003.03600/) (2020)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.07169/) (2020)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research](https://mlsanity.com/paper/1903.00742/) (2019)
- [Off-Policy Actor-Critic with Shared Experience Replay](https://mlsanity.com/paper/1909.11583/) (2019)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [WordCraft: An Environment for Benchmarking Commonsense Agents](https://mlsanity.com/paper/2007.09185/) (2020)
- [Superstition in the Network: Deep Reinforcement Learning Plays Deceptive Games](https://mlsanity.com/paper/1908.04436/) (2019)
- [Reinforcement Communication Learning in Different Social Network Structures](https://mlsanity.com/paper/2007.09820/) (2020)
- [Experience Sharing Between Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/1911.02191/) (2019)

## Built on by (most cited first, 10 of 10)

- [A social path to human-like artificial intelligence](https://mlsanity.com/paper/2405.15815/) (2024)
- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [Cultural evolution in populations of Large Language Models](https://mlsanity.com/paper/2403.08882/) (2024)
- [Collective Innovation in Groups of Large Language Models](https://mlsanity.com/paper/2407.05377/) (2024)
- [A Rubric for Human-like Agents and NeuroAI](https://mlsanity.com/paper/2212.04401/) (2022)
- [Memory, Space, and Planning: Multiscale Predictive Representations](https://mlsanity.com/paper/2401.09491/) (2024)
- [Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?](https://mlsanity.com/paper/2504.03814/) (2025)
- [A Compositional Framework for Open-ended Intelligence](https://mlsanity.com/paper/2606.15386/) (2026)
- [High-fidelity social learning via shared episodic memories enhances collaborative foraging through mnemonic convergence](https://mlsanity.com/paper/2412.20271/) (2024)
- [Discovering Adaptive Transmission Programs for Collective Innovation](https://mlsanity.com/paper/2608.24545/) (2026)

## Abstract (arXiv)

Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective search and accumulation. Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of social network structure on innovation has not been systematically studied in RL. Here, we use a multi-level problem setting (WordCraft), with three different innovation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms. We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully-connected, small world, dynamic, ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward-focused evaluation framework of RL. Comparing the level of innovation achieved by different social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation in tasks with a deceptive nature and large search spaces. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, the metrics we propose, can help understand the success of different social network structures on different tasks, with the diversity of experiences on an individual and group level lending crucial insights.