mlsanityData to 14 Oct 2026

An Optimal Policy for Learning Controllable Dynamics by Exploration

arXiv 2512.20053 ↗First posted 23 Dec 2025cs.LG cs.AI cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Controllable Markov chains describe the dynamics of sequential decision making tasks and are the central component in optimal control and reinforcement learning. In this work, we give the general form of an optimal policy for learning controllable dynamics in an unknown environment by exploring over a limited time horizon. This policy is simple to implement and efficient to compute, and allows an agent to ``learn by exploring" as it maximizes its information gain in a greedy fashion by selecting controls from a constraint set that changes over time during exploration. We give a simple parameterization for the set of controls, and present an algorithm for finding an optimal policy. The reason for this policy is due to the existence of certain types of states that restrict control of the dynamics; such as transient states, absorbing states, and non-backtracking states. We show why the occurrence of these states makes a non-stationary policy essential for achieving optimal exploration. Six interesting examples of controllable dynamics are treated in detail. Policy optimality is demonstrated using counting arguments, comparing with suboptimal policies, and by making use of a sequential improvement property from dynamic programming.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 3 of its 19 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

3 of 19 references.

References · 3

  1. 2021A Dynamic Programming Algorithm for Finding an Optimal Sequence of Informative Measurementscited by 3
  2. 2025On-line Policy Improvement using Monte-Carlo Searchcited by 3
  3. 2025Learning controllable dynamics through informative explorationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  2. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
  3. 2024Model-Free Active Exploration in Reinforcement Learning
  4. 2026Learning in Markov Decision Processes with Exogenous Dynamics
  5. 2024Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction
  6. 2026Average-Reward Policy Identification from a Single Continuing Trajectory
  7. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  8. 2025Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective