# Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings

arXiv: https://arxiv.org/abs/2610.03065 · HTML: https://mlsanity.com/paper/2610.03065/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 29 of 29)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Generalized Teacher Forcing for Learning Chaotic Dynamics](https://mlsanity.com/paper/2306.04406/) (2023)
- [Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI](https://mlsanity.com/paper/1902.07186/) (2019)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processes](https://mlsanity.com/paper/1706.06544/) (2017)
- [A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements](https://mlsanity.com/paper/1612.07846/) (2016)
- [Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2410.14240/) (2024)
- [Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems](https://mlsanity.com/paper/2111.01256/) (2021)
- [Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data](https://mlsanity.com/paper/2410.04814/) (2024)
- [Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics](https://mlsanity.com/paper/2212.07892/) (2022)
- [Meta-Dynamical State Space Models for Integrative Neural Data Analysis](https://mlsanity.com/paper/2410.05454/) (2024)
- [Combining Deep Reinforcement Learning And Local Control For The Acrobot Swing-up And Balance Task](https://mlsanity.com/paper/2012.11663/) (2020)
- [Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs](https://mlsanity.com/paper/2506.07919/) (2025)
- [Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling](https://mlsanity.com/paper/2602.16864/) (2026)
- [Neurophysiologically Realistic Environment for Comparing Adaptive Deep Brain Stimulation Algorithms in Parkinson Disease](https://mlsanity.com/paper/2505.09624/) (2025)
- [Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease](https://mlsanity.com/paper/2507.06326/) (2025)

## Abstract (arXiv)

Learning control from action-free recordings is challenging because intervention effects are unobserved and policies may exploit errors in reconstructed dynamics. We present a hierarchical model-based reinforcement learning framework that uses shared structure across related systems to learn system-specific control policies from action-free recordings. A hierarchical dynamical system reconstruction model captures shared dynamics and individual variation through low-dimensional embeddings. These embeddings are then reused to parameterize shared policy and value networks, linking differences in reconstructed dynamics to differences in control. Policies are trained entirely via simulation under an explicit intervention model with additive latent perturbations. Piecewise-linear recurrent neural networks enable mechanistic analyses of the controlled dynamics, while decoder-based constraints make the immediate effects of interventions interpretable in observation space and permit interventions on one modality while protecting another from direct manipulation. On Lorenz-63 and double-pendulum systems, hierarchical policies improve transfer over independently trained policies. On Lorenz-63, they also achieve a higher mean reward than repeated planning with the same reconstructed models, perform comparably to methods trained with controlled interactions, and generalize to systems absent from policy training after embedding inference alone. Applications to neural-behavioral recordings demonstrate suppression of predicted movement under constrained neural perturbations. Together, these findings show how shared dynamical representations support transferable control and mechanistic hypothesis generation from action-free recordings.