# Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2401.05821 · HTML: https://mlsanity.com/paper/2401.05821/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 70 of its 110 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 30 of 70)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Making deep neural networks right for the right scientific reasons by interacting with their explanations](https://mlsanity.com/paper/2001.05371/) (2020)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities](https://mlsanity.com/paper/2111.06420/) (2021)

## Built on by (most cited first, 23 of 23)

- [Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning](https://mlsanity.com/paper/2405.14956/) (2024)
- [End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations](https://mlsanity.com/paper/2403.12451/) (2024)
- [HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning](https://mlsanity.com/paper/2406.03997/) (2024)
- [Object Centric Concept Bottlenecks](https://mlsanity.com/paper/2505.24492/) (2025)
- [Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization](https://mlsanity.com/paper/2408.08761/) (2024)
- [Interpretable end-to-end Neurosymbolic Reinforcement Learning agents](https://mlsanity.com/paper/2410.14371/) (2024)
- [Evaluating Interpretable Reinforcement Learning by Distilling Policies into Programs](https://mlsanity.com/paper/2503.08322/) (2025)
- [Better Decisions through the Right Causal World Model](https://mlsanity.com/paper/2504.07257/) (2025)
- [Nonparametric Identification of Latent Concepts](https://mlsanity.com/paper/2510.00136/) (2025)
- [Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations](https://mlsanity.com/paper/2407.20651/) (2024)
- [Deep Reinforcement Learning via Object-Centric Attention](https://mlsanity.com/paper/2504.03024/) (2025)
- [Gradient Boosting Reinforcement Learning](https://mlsanity.com/paper/2407.08250/) (2024)
- [Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning](https://mlsanity.com/paper/2501.17077/) (2025)
- [Towards Automated Semantic Interpretability in Reinforcement Learning via Vision-Language Models](https://mlsanity.com/paper/2503.16724/) (2025)
- [Human-Allied Relational Reinforcement Learning](https://mlsanity.com/paper/2510.16188/) (2025)
- [Selecting Decision-Relevant Concepts in Reinforcement Learning](https://mlsanity.com/paper/2604.04808/) (2026)
- [Towards a Research Community in Interpretable Reinforcement Learning: the InterpPol Workshop](https://mlsanity.com/paper/2404.10906/) (2024)
- [OCALM: Object-Centric Assessment with Language Models](https://mlsanity.com/paper/2406.16748/) (2024)
- [Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learning](https://mlsanity.com/paper/2604.02353/) (2026)
- [Unveiling Complex Collective Behaviors from Simple Rewards](https://mlsanity.com/paper/2607.12861/) (2026)
- [From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems](https://mlsanity.com/paper/2607.15459/) (2026)
- [Escaping Local Views: Discovering Latent Concepts for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.34459/) (2026)
- [VERA: Verifiable Feasibility Representations with Counterfactual Credit for Constrained Multi-Agent Control](https://mlsanity.com/paper/2609.38889/) (2026)

## Abstract (arXiv)

Goal misalignment, reward sparsity and difficult credit assignment are only a few of the many issues that make it difficult for deep reinforcement learning (RL) agents to learn optimal policies. Unfortunately, the black-box nature of deep neural networks impedes the inclusion of domain experts for inspecting the model and revising suboptimal policies. To this end, we introduce *Successive Concept Bottleneck Agents* (SCoBots), that integrate consecutive concept bottleneck (CB) layers. In contrast to current CB models, SCoBots do not just represent concepts as properties of individual objects, but also as relations between objects which is crucial for many RL tasks. Our experimental results provide evidence of SCoBots' competitive performances, but also of their potential for domain experts to understand and regularize their behavior. Among other things, SCoBots enabled us to identify a previously unknown misalignment problem in the iconic video game, Pong, and resolve it. Overall, SCoBots thus result in more human-aligned RL agents. Our code is available at https://github.com/k4ntz/SCoBots .