mlsanityData to 7 Oct 2026

Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents

arXiv 2401.05821 ↗First posted 11 Jan 2024cs.LG cs.SCcited by 23

Not on the map. Why?

Abstract (arXiv)

Goal misalignment, reward sparsity and difficult credit assignment are only a few of the many issues that make it difficult for deep reinforcement learning (RL) agents to learn optimal policies. Unfortunately, the black-box nature of deep neural networks impedes the inclusion of domain experts for inspecting the model and revising suboptimal policies. To this end, we introduce *Successive Concept Bottleneck Agents* (SCoBots), that integrate consecutive concept bottleneck (CB) layers. In contrast to current CB models, SCoBots do not just represent concepts as properties of individual objects, but also as relations between objects which is crucial for many RL tasks. Our experimental results provide evidence of SCoBots' competitive performances, but also of their potential for domain experts to understand and regularize their behavior. Among other things, SCoBots enabled us to identify a previously unknown misalignment problem in the iconic video game, Pong, and resolve it. Overall, SCoBots thus result in more human-aligned RL agents. Our code is available at https://github.com/k4ntz/SCoBots .

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 70 of its 110 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

70 of 110 references.

References · 70

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2020Concept Bottleneck Modelscited by 1,022
  11. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  12. 2020Object-Centric Learning with Slot Attentioncited by 865
38 more
  1. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  4. 2019Model-Based Reinforcement Learning for Ataricited by 495
  5. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  6. 2016The Option-Critic Architecturecited by 433
  7. 2018AI safety via debatecited by 384
  8. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2023Reward Design with Language Modelscited by 257
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  12. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  13. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  14. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  15. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  16. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  17. 2020Explainable Deep Learning: A Field Guide for the Uninitiatedcited by 127
  18. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  19. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  20. 2020Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanationscited by 94
  21. 2022Concept Bottleneck Model with Additional Unsupervised Conceptscited by 79
  22. 2022Interactive Concept Bottleneck Modelscited by 79
  23. 2022GlanceNets: Interpretabile, Leak-proof Concept-based Modelscited by 73
  24. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
  25. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  26. 2019Neural Logic Reinforcement Learningcited by 53
  27. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  28. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
  29. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  30. 2023Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstractioncited by 45
  31. 2022A Comparative Study of Faithfulness Metrics for Model Interpretability Methodscited by 38
  32. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  33. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 33
  34. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  35. 2021Neuro-Symbolic Reinforcement Learning with First-Order Logiccited by 30
  36. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  37. 2023Learning to Intervene on Concept Bottleneckscited by 29
  38. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 8; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
  2. 2024End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanationscited by 17
  3. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
  4. 2025Object Centric Concept Bottleneckscited by 10
  5. 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimizationcited by 5
  6. 2024Interpretable end-to-end Neurosymbolic Reinforcement Learning agentscited by 4
  7. 2025Evaluating Interpretable Reinforcement Learning by Distilling Policies into Programscited by 3
  8. 2025Better Decisions through the Right Causal World Modelcited by 3
  9. 2025Nonparametric Identification of Latent Conceptscited by 3
  10. 2024Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representationscited by 2
  11. 2025Deep Reinforcement Learning via Object-Centric Attentioncited by 2
  12. 2024Gradient Boosting Reinforcement Learningcited by 1
11 more
  1. 2025Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learningcited by 1
  2. 2025Towards Automated Semantic Interpretability in Reinforcement Learning via Vision-Language Modelscited by 1
  3. 2025Human-Allied Relational Reinforcement Learningcited by 1
  4. 2026Selecting Decision-Relevant Concepts in Reinforcement Learningcited by 1
  5. 2024Towards a Research Community in Interpretable Reinforcement Learning: the InterpPol Workshopcited by 0
  6. 2024OCALM: Object-Centric Assessment with Language Modelscited by 0
  7. 2026Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learningcited by 0
  8. 2026Unveiling Complex Collective Behaviors from Simple Rewardscited by 0
  9. 2026From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systemscited by 0
  10. 2026Escaping Local Views: Discovering Latent Concepts for Interpretable Multi-Agent Reinforcement Learningcited by 0
  11. 2026VERA: Verifiable Feasibility Representations with Counterfactual Credit for Constrained Multi-Agent Controlcited by 0