# Learning low-dimensional state embeddings and metastable clusters from time series data

arXiv: https://arxiv.org/abs/1906.00302 · HTML: https://mlsanity.com/paper/1906.00302/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2019.

## Built on by (most cited first, 6 of 6)

- [Optimizing Automated Picking Systems in Warehouse Robots Using Machine Learning](https://mlsanity.com/paper/2408.16633/) (2024)
- [Deep Reinforcement Learning-based Obstacle Avoidance for Robot Movement in Warehouse Environments](https://mlsanity.com/paper/2409.14972/) (2024)
- [Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout](https://mlsanity.com/paper/2411.06128/) (2024)
- [Designing Long-term Group Fair Policies in Dynamical Systems](https://mlsanity.com/paper/2311.12447/) (2023)
- [Learning Good State and Action Representations via Tensor Decomposition](https://mlsanity.com/paper/2105.01136/) (2021)
- [In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates](https://mlsanity.com/paper/2410.15612/) (2024)

## Abstract (arXiv)

This paper studies how to find compact state embeddings from high-dimensional Markov state trajectories, where the transition kernel has a small intrinsic rank. In the spirit of diffusion map, we propose an efficient method for learning a low-dimensional state embedding and capturing the process's dynamics. This idea also leads to a kernel reshaping method for more accurate nonparametric estimation of the transition function. State embedding can be used to cluster states into metastable sets, thereby identifying the slow dynamics. Sharp statistical error bounds and misclassification rate are proved. Experiment on a simulated dynamical system shows that the state clustering method indeed reveals metastable structures. We also experiment with time series generated by layers of a Deep-Q-Network when playing an Atari game. The embedding method identifies game states to be similar if they share similar future events, even though their raw data are far different.