# The Frost Hollow Experiments: Pavlovian Signalling as a Path to Coordination and Communication Between Agents

arXiv: https://arxiv.org/abs/2203.09498 · HTML: https://mlsanity.com/paper/2203.09498/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 86 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences](https://mlsanity.com/paper/1610.09513/) (2016)
- [A Clockwork RNN](https://mlsanity.com/paper/1402.3511/) (2014)
- [Emergence of grid-like representations by training recurrent neural networks to perform spatial localization](https://mlsanity.com/paper/1803.07770/) (2018)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [Cooperating with Machines](https://mlsanity.com/paper/1703.06207/) (2017)
- [Launchpad: A Programming Model for Distributed Machine Learning Research](https://mlsanity.com/paper/2106.04516/) (2021)
- [From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learning](https://mlsanity.com/paper/2011.04590/) (2020)
- [Learned human-agent decision-making, communication and joint action in a virtual reality environment](https://mlsanity.com/paper/1905.02691/) (2019)
- [Assessing Human Interaction in Virtual Reality With Continually Learning Prediction Agents Based on Reinforcement Learning Algorithms: A Pilot Study](https://mlsanity.com/paper/2112.07774/) (2021)
- [Communicative Capital for Prosthetic Agents](https://mlsanity.com/paper/1711.03676/) (2017)
- [Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making](https://mlsanity.com/paper/2201.03709/) (2022)

## Built on by (most cited first, 2 of 2)

- [The Alberta Plan for AI Research](https://mlsanity.com/paper/2208.11173/) (2022)
- [Continually Learned Pavlovian Signalling Without Forgetting for Human-in-the-Loop Robotic Control](https://mlsanity.com/paper/2305.14365/) (2023)

## Abstract (arXiv)

Learned communication between agents is a powerful tool when approaching decision-making problems that are hard to overcome by any single agent in isolation. However, continual coordination and communication learning between machine agents or human-machine partnerships remains a challenging open problem. As a stepping stone toward solving the continual communication learning problem, in this paper we contribute a multi-faceted study into what we term Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent with different perceptual access to their shared environment. We seek to establish how different temporal processes and representational choices impact Pavlovian signalling between learning agents. To do so, we introduce a partially observable decision-making domain we call the Frost Hollow. In this domain a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that seeks to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: 1) machine prediction and control learning in a linear walk, and 2) a prediction learning machine interacting with a human participant in a virtual reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning. Our results therefore point to an actionable, constructivist path towards continual communication learning between reinforcement learning agents, with potential impact in a range of real-world settings.