# Dynamical Priors as a Training Objective in Reinforcement Learning

arXiv: https://arxiv.org/abs/2604.21464 · HTML: https://mlsanity.com/paper/2604.21464/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 6 of 6)

- [Simplified Temporal Consistency Reinforcement Learning](https://mlsanity.com/paper/2306.09466/) (2023)
- [Real-Time Recurrent Reinforcement Learning](https://mlsanity.com/paper/2311.04830/) (2023)
- [Unifying and generalizing models of neural dynamics during decision-making](https://mlsanity.com/paper/2001.04571/) (2020)
- [Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.08794/) (2023)
- [Multi-timescale memory dynamics in a reinforcement learning network with attention-gated memory](https://mlsanity.com/paper/1712.10062/) (2017)
- [Deep reinforcement learning with time-scale invariant memory](https://mlsanity.com/paper/2412.15292/) (2024)

## Abstract (arXiv)

Standard reinforcement learning (RL) optimizes policies for reward but imposes few constraints on how decisions evolve over time. As a result, policies may achieve high performance while exhibiting temporally incoherent behavior such as abrupt confidence shifts, oscillations, or degenerate inactivity. We introduce Dynamical Prior Reinforcement Learning (DP-RL), a training framework that augments policy gradient learning with an auxiliary loss derived from external state dynamics that implement evidence accumulation and hysteresis. Without modifying the reward, environment, or policy architecture, this prior shapes the temporal evolution of action probabilities during learning. Across three minimal environments, we show that dynamical priors systematically alter decision trajectories in task-dependent ways, promoting temporally structured behavior that cannot be explained by generic smoothing. These results demonstrate that training objectives alone can control the temporal geometry of decision-making in RL agents.