# Enhancing Hierarchical Reinforcement Learning through Change Point Detection in Time Series

arXiv: https://arxiv.org/abs/2510.24988 · HTML: https://mlsanity.com/paper/2510.24988/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding](https://mlsanity.com/paper/2011.14097/) (2020)
- [Multi-Level Discovery of Deep Options](https://mlsanity.com/paper/1703.08294/) (2017)
- [Learning Abstract Options](https://mlsanity.com/paper/1810.11583/) (2018)
- [The Termination Critic](https://mlsanity.com/paper/1902.09996/) (2019)
- [Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detection](https://mlsanity.com/paper/2105.09452/) (2021)

## Abstract (arXiv)

Hierarchical Reinforcement Learning (HRL) enhances the scalability of decision-making in long-horizon tasks by introducing temporal abstraction through options-policies that span multiple timesteps. Despite its theoretical appeal, the practical implementation of HRL suffers from the challenge of autonomously discovering semantically meaningful subgoals and learning optimal option termination boundaries. This paper introduces a novel architecture that integrates a self-supervised, Transformer-based Change Point Detection (CPD) module into the Option-Critic framework, enabling adaptive segmentation of state trajectories and the discovery of options. The CPD module is trained using heuristic pseudo-labels derived from intrinsic signals to infer latent shifts in environment dynamics without external supervision. These inferred change-points are leveraged in three critical ways: (i) to serve as supervisory signals for stabilizing termination function gradients, (ii) to pretrain intra-option policies via segment-wise behavioral cloning, and (iii) to enforce functional specialization through inter-option divergence penalties over CPD-defined state partitions. The overall optimization objective enhances the standard actor-critic loss using structure-aware auxiliary losses. In our framework, option discovery arises naturally as CPD-defined trajectory segments are mapped to distinct intra-option policies, enabling the agent to autonomously partition its behavior into reusable, semantically meaningful skills. Experiments on the Four-Rooms and Pinball tasks demonstrate that CPD-guided agents exhibit accelerated convergence, higher cumulative returns, and significantly improved option specialization. These findings confirm that integrating structural priors via change-point segmentation leads to more interpretable, sample-efficient, and robust hierarchical policies in complex environments.