# Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering

arXiv: https://arxiv.org/abs/2409.17411 · HTML: https://mlsanity.com/paper/2409.17411/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild](https://mlsanity.com/paper/2304.00451/) (2023)
- [Towards Interpretable Reinforcement Learning Using Attention Augmented Agents](https://mlsanity.com/paper/1906.02500/) (2019)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Graying the black box: Understanding DQNs](https://mlsanity.com/paper/1602.02658/) (2016)
- [word2vec Parameter Learning Explained](https://mlsanity.com/paper/1411.2738/) (2014)
- [Improving Unsupervised Image Clustering With Robust Learning](https://mlsanity.com/paper/2012.11150/) (2020)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [Towards Robust Bisimulation Metric Learning](https://mlsanity.com/paper/2110.14096/) (2021)
- [Vector Quantized Models for Planning](https://mlsanity.com/paper/2106.04615/) (2021)
- [Discriminatively Boosted Image Clustering with Fully Convolutional Auto-Encoders](https://mlsanity.com/paper/1703.07980/) (2017)
- [Concept Learning for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.12232/) (2023)
- [Smaller World Models for Reinforcement Learning](https://mlsanity.com/paper/2010.05767/) (2020)

## Built on by (most cited first, 1 of 1)

- [SAT-RTS: A systematic framework for tactical knowledge extraction and visualization-based analysis in real-time strategy games](https://mlsanity.com/paper/2606.30090/) (2026)

## Abstract (arXiv)

In this paper, we explore semantic clustering properties of deep reinforcement learning (DRL) to improve its interpretability and deepen our understanding of its internal semantic organization. In this context, semantic clustering refers to the ability of neural networks to cluster inputs based on their semantic similarity in the feature space. We propose a DRL architecture that incorporates a novel semantic clustering module that combines feature dimensionality reduction with online clustering. This module integrates seamlessly into the DRL training pipeline, addressing the instability of t-SNE and eliminating the need for extensive manual annotation inherent to prior semantic analysis methods. We experimentally validate the effectiveness of the proposed module and demonstrate its ability to reveal semantic clustering properties within DRL. Furthermore, we introduce new analytical methods based on these properties to provide insights into the hierarchical structure of policies and semantic organization within the feature space. Our code is available at https://github.com/ualiangzhang/semantic_rl.