# Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability

arXiv: https://arxiv.org/abs/2109.07259 · HTML: https://mlsanity.com/paper/2109.07259/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 87 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 5 of 5)

- [Deep reinforcement learning in World-Earth system models to discover sustainable management strategies](https://mlsanity.com/paper/1908.05567/) (2019)
- [Deterministic limit of temporal difference reinforcement learning for stochastic games](https://mlsanity.com/paper/1809.07225/) (2018)
- [Timing uncertainty in collective risk dilemmas encourages group reciprocation and polarization](https://mlsanity.com/paper/2003.07317/) (2020)
- [Punishment and inspection for governing the commons in a feedback-evolving game](https://mlsanity.com/paper/1807.05484/) (2018)
- [Environmental feedback drives cooperation in spatial social dilemmas](https://mlsanity.com/paper/1802.06745/) (2018)

## Built on by (most cited first, 3 of 3)

- [Intrinsic fluctuations of reinforcement learning promote cooperation](https://mlsanity.com/paper/2209.01013/) (2022)
- [K-Level Policy Gradients for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.12117/) (2025)
- [Learned Collusion](https://mlsanity.com/paper/2304.12647/) (2023)

## Abstract (arXiv)

Assessing the systemic effects of uncertainty that arises from agents' partial observation of the true states of the world is critical for understanding a wide range of scenarios. Yet, previous modeling work on agent learning and decision-making either lacks a systematic way to describe this source of uncertainty or puts the focus on obtaining optimal policies using complex models of the world that would impose an unrealistically high cognitive demand on real agents. In this work we aim to efficiently describe the emergent behavior of biologically plausible and parsimonious learning agents faced with partially observable worlds. Therefore we derive and present deterministic reinforcement learning dynamics where the agents observe the true state of the environment only partially. We showcase the broad applicability of our dynamics across different classes of partially observable agent-environment systems. We find that partial observability creates unintuitive benefits in a number of specific contexts, pointing the way to further research on a general understanding of such effects. For instance, partially observant agents can learn better outcomes faster, in a more stable way and even overcome social dilemmas. Furthermore, our method allows the application of dynamical systems theory to partially observable multiagent leaning. In this regard we find the emergence of catastrophic limit cycles, a critical slowing down of the learning processes between reward regimes and the separation of the learning dynamics into fast and slow directions, all caused by partial observability. Therefore, the presented dynamics have the potential to become a formal, yet practical, lightweight and robust tool for researchers in biology, social science and machine learning to systematically investigate the effects of interacting partially observant agents.