# Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making

arXiv: https://arxiv.org/abs/2201.03709 · HTML: https://mlsanity.com/paper/2201.03709/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 11 Jan 2022.

## Builds on (most cited first, 10 of 10)

- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences](https://mlsanity.com/paper/1610.09513/) (2016)
- [A Clockwork RNN](https://mlsanity.com/paper/1402.3511/) (2014)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [Cooperating with Machines](https://mlsanity.com/paper/1703.06207/) (2017)
- [Launchpad: A Programming Model for Distributed Machine Learning Research](https://mlsanity.com/paper/2106.04516/) (2021)
- [From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learning](https://mlsanity.com/paper/2011.04590/) (2020)
- [Learned human-agent decision-making, communication and joint action in a virtual reality environment](https://mlsanity.com/paper/1905.02691/) (2019)
- [Assessing Human Interaction in Virtual Reality With Continually Learning Prediction Agents Based on Reinforcement Learning Algorithms: A Pilot Study](https://mlsanity.com/paper/2112.07774/) (2021)
- [Communicative Capital for Prosthetic Agents](https://mlsanity.com/paper/1711.03676/) (2017)

## Built on by (most cited first, 2 of 2)

- [The Frost Hollow Experiments: Pavlovian Signalling as a Path to Coordination and Communication Between Agents](https://mlsanity.com/paper/2203.09498/) (2022)
- [What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience](https://mlsanity.com/paper/2206.06485/) (2022)

## Abstract (arXiv)

In this paper, we contribute a multi-faceted study into Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent. Signalling is intimately connected to time and timing. In service of generating and receiving signals, humans and other animals are known to represent time, determine time since past events, predict the time until a future stimulus, and both recognize and generate patterns that unfold in time. We investigate how different temporal processes impact coordination and signalling between learning agents by introducing a partially observable decision-making domain we call the Frost Hollow. In this domain, a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that works to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: machine agents interacting in a seven-state linear walk, and human-machine interaction in a virtual-reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning between two agents. We further show how to computationally build this adaptive signalling process out of a fixed signalling process, characterized by fast continual prediction learning and minimal constraints on the nature of the agent receiving signals. Our results therefore suggest an actionable, constructivist path towards communication learning between reinforcement learning agents.