# Hierarchical State Abstraction Based on Structural Information Principles

arXiv: https://arxiv.org/abs/2304.12000 · HTML: https://mlsanity.com/paper/2304.12000/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Near Optimal Behavior via Approximate State Abstraction](https://mlsanity.com/paper/1701.04113/) (2017)
- [Structural Entropy Guided Graph Hierarchical Pooling](https://mlsanity.com/paper/2206.13510/) (2022)
- [Learning Markov State Abstractions for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04379/) (2021)
- [Reinforced Neighborhood Selection Guided Multi-Relational Graph Neural Networks](https://mlsanity.com/paper/2104.07886/) (2021)
- [SE-GSL: A General and Effective Graph Structure Learning Framework through Structural Entropy Optimization](https://mlsanity.com/paper/2303.09778/) (2023)
- [SimSR: Simple Distance-based State Representation for Deep Reinforcement Learning](https://mlsanity.com/paper/2112.15303/) (2021)
- [A Simple yet Effective Method for Graph Classification](https://mlsanity.com/paper/2206.02404/) (2022)
- [Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles](https://mlsanity.com/paper/2304.00755/) (2023)
- [USER: Unsupervised Structural Entropy-based Robust Graph Neural Network](https://mlsanity.com/paper/2302.05889/) (2023)
- [Automating DBSCAN via Deep Reinforcement Learning](https://mlsanity.com/paper/2208.04537/) (2022)

## Built on by (most cited first, 14 of 14)

- [SeBot: Structural Entropy Guided Multi-View Contrastive Learning for Social Bot Detection](https://mlsanity.com/paper/2405.11225/) (2024)
- [MultiSPANS: A Multi-range Spatial-Temporal Transformer Network for Traffic Forecast via Structural Entropy Optimization](https://mlsanity.com/paper/2311.02880/) (2023)
- [Adversarial Socialbots Modeling Based on Structural Information Principles](https://mlsanity.com/paper/2312.08098/) (2023)
- [Unsupervised Skin Lesion Segmentation via Structural Entropy Minimization on Multi-Scale Superpixel Graphs](https://mlsanity.com/paper/2309.01899/) (2023)
- [Effective Exploration Based on the Structural Information Principles](https://mlsanity.com/paper/2410.06621/) (2024)
- [Unsupervised Graph Clustering with Deep Structural Entropy](https://mlsanity.com/paper/2505.14040/) (2025)
- [KAT-Coder-V2.5 Technical Report](https://mlsanity.com/paper/2607.05471/) (2026)
- [State Chrono Representation for Enhancing Generalization in Reinforcement Learning](https://mlsanity.com/paper/2411.06174/) (2024)
- [Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning](https://mlsanity.com/paper/2509.21942/) (2025)
- [Hyperbolic Continuous Structural Entropy for Hierarchical Clustering](https://mlsanity.com/paper/2512.00524/) (2025)
- [RoBCtrl: Attacking GNN-Based Social Bot Detectors via Reinforced Manipulation of Bots Control Interaction](https://mlsanity.com/paper/2510.16035/) (2025)
- [Adaptive and Robust DBSCAN with Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2505.04339/) (2025)
- [Structure-Induced Information for Rerooting Levin Tree Search](https://mlsanity.com/paper/2605.30664/) (2026)
- [SAT-RTS: A systematic framework for tactical knowledge extraction and visualization-based analysis in real-time strategy games](https://mlsanity.com/paper/2606.30090/) (2026)

## Abstract (arXiv)

State abstraction optimizes decision-making by ignoring irrelevant environmental information in reinforcement learning with rich observations. Nevertheless, recent approaches focus on adequate representational capacities resulting in essential information loss, affecting their performances on challenging tasks. In this article, we propose a novel mathematical Structural Information principles-based State Abstraction framework, namely SISA, from the information-theoretic perspective. Specifically, an unsupervised, adaptive hierarchical state clustering method without requiring manual assistance is presented, and meanwhile, an optimal encoding tree is generated. On each non-root tree node, a new aggregation function and condition structural entropy are designed to achieve hierarchical state abstraction and compensate for sampling-induced essential information loss in state abstraction. Empirical evaluations on a visual gridworld domain and six continuous control benchmarks demonstrate that, compared with five SOTA state abstraction approaches, SISA significantly improves mean episode reward and sample efficiency up to 18.98 and 44.44%, respectively. Besides, we experimentally show that SISA is a general framework that can be flexibly integrated with different representation-learning objectives to improve their performances further.