# A Novel Multi-Timescale Stability-Preserving Hierarchical Reinforcement Learning Controller Framework for Adaptive Control in High-Dimensional Dynamical Systems

arXiv: https://arxiv.org/abs/2510.22420 · HTML: https://mlsanity.com/paper/2510.22420/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 59 references that are in the map.
- First posted on arXiv 25 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning](https://mlsanity.com/paper/2007.08459/) (2020)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [Making Linear MDPs Practical via Contrastive Representation Learning](https://mlsanity.com/paper/2207.07150/) (2022)
- [Actor-Critic Reinforcement Learning for Control with Stability Guarantee](https://mlsanity.com/paper/2004.14288/) (2020)
- [Data-Driven Reduced-Order Modeling of Spatiotemporal Chaos with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2109.00060/) (2021)
- [A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG](https://mlsanity.com/paper/2503.24307/) (2025)
- [Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach](https://mlsanity.com/paper/2304.04066/) (2023)
- [Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning](https://mlsanity.com/paper/2405.09660/) (2024)
- [Continuous-Time State Estimation Methods in Robotics: A Survey](https://mlsanity.com/paper/2411.03951/) (2024)
- [Ultra High-Dimensional Nonlinear Feature Selection for Big Biological Data](https://mlsanity.com/paper/1608.04048/) (2016)
- [Time Series Embedding Methods for Classification Tasks: A Review](https://mlsanity.com/paper/2501.13392/) (2025)
- [Risk-Aware Robotics: Tail Risk Measures in Planning, Control, and Verification](https://mlsanity.com/paper/2403.18972/) (2024)
- [Optimal Inferential Control of Convolutional Neural Networks](https://mlsanity.com/paper/2410.09663/) (2024)
- [Optimizing Flamelet Generated Manifold Models: A Machine Learning Performance Study](https://mlsanity.com/paper/2507.01030/) (2025)
- [Interpretable Machine Learning for Predicting Startup Funding, Patenting, and Exits](https://mlsanity.com/paper/2510.09465/) (2025)
- [Exploring reinforcement learning techniques for discrete and continuous control tasks in the MuJoCo environment](https://mlsanity.com/paper/2307.11166/) (2023)

## Abstract (arXiv)

Controlling high-dimensional stochastic systems, critical in robotics, autonomous vehicles, and hyperchaotic systems, faces the curse of dimensionality, lacks temporal abstraction, and often fails to ensure stochastic stability. To overcome these limitations, this study introduces the Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning (MTLHRL) framework. MTLHRL integrates a hierarchical policy within a semi-Markov Decision Process (SMDP), featuring a high-level policy for strategic planning and a low-level policy for reactive control, which effectively manages complex, multi-timescale decision-making and reduces dimensionality overhead. Stability is rigorously enforced using a neural Lyapunov function optimized via Lagrangian relaxation and multi-timescale actor-critic updates, ensuring mean-square boundedness or asymptotic stability in the face of stochastic dynamics. The framework promotes efficient and reliable learning through trust-region constraints and decoupled optimization. Extensive simulations on an 8D hyperchaotic system and a 5-DOF robotic manipulator demonstrate MTLHRL's empirical superiority. It significantly outperforms baseline methods in both stability and performance, recording the lowest error indices (e.g., Integral Absolute Error (IAE): 3.912 in hyperchaotic control and IAE: 1.623 in robotics), achieving faster convergence, and exhibiting superior disturbance rejection. MTLHRL offers a theoretically grounded and practically viable solution for robust control of complex stochastic systems.