# Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2307.12143 · HTML: https://mlsanity.com/paper/2307.12143/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Attractor and integrator networks in the brain](https://mlsanity.com/paper/2112.03978/) (2021)
- [Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics](https://mlsanity.com/paper/1906.10720/) (2019)
- [Perspective Taking in Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1907.01851/) (2019)
- [APES: a Python toolbox for simulating reinforcement learning environments](https://mlsanity.com/paper/1808.10692/) (2018)

## Built on by (most cited first, 1 of 1)

- [On-Demand Model and Client Deployment in Federated Learning with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.07175/) (2024)

## Abstract (arXiv)

Adapting to regularities of the environment is critical for biological organisms to anticipate events and plan. A prominent example is the circadian rhythm corresponding to the internalization by organisms of the 24-hour period of the Earth's rotation. In this work, we study the emergence of circadian-like rhythms in deep reinforcement learning agents. In particular, we deployed agents in an environment with a reliable periodic variation while solving a foraging task. We systematically characterize the agent's behavior during learning and demonstrate the emergence of a rhythm that is endogenous and entrainable. Interestingly, the internal rhythm adapts to shifts in the phase of the environmental signal without any re-training. Furthermore, we show via bifurcation and phase response curve analyses how artificial neurons develop dynamics to support the internalization of the environmental rhythm. From a dynamical systems view, we demonstrate that the adaptation proceeds by the emergence of a stable periodic orbit in the neuron dynamics with a phase response that allows an optimal phase synchronisation between the agent's dynamics and the environmental rhythm.