mlsanityData to 14 Oct 2026

Interpretable end-to-end Neurosymbolic Reinforcement Learning agents

arXiv 2410.14371 ↗First posted 18 Oct 2024cs.AIcited by 4

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

Deep reinforcement learning (RL) agents rely on shortcut learning, preventing them from generalizing to slightly different environments. To address this problem, symbolic method, that use object-centric states, have been developed. However, comparing these methods to deep agents is not fair, as these last operate from raw pixel-based states. In this work, we instantiate the symbolic SCoBots framework. SCoBots decompose RL tasks into intermediate, interpretable representations, culminating in action decisions based on a comprehensible set of object-centric relational concepts. This architecture aids in demystifying agent decisions. By explicitly learning to extract object-centric representations from raw states, object-centric RL, and policy distillation via rule extraction, this work places itself within the neurosymbolic AI paradigm, blending the strengths of neural networks with symbolic AI. We present the first implementation of an end-to-end trained SCoBot, separately evaluate of its components, on different Atari games. The results demonstrate the framework's potential to create interpretable and performing RL systems, and pave the way for future research directions in obtaining end-to-end interpretable RL agents.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 50 of its 59 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

50 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024EXPIL: Explanatory Predicate Invention for Learning in Games10 Jun
  2. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning23 May
  3. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents11 Jan

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2020Concept Bottleneck Modelscited by 1,032
  7. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  8. 2020Object-Centric Learning with Slot Attentioncited by 876
  9. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  10. 2015Policy Distillationcited by 440
  11. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 425
  12. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
35 more
  1. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  2. 2021Conditional Object-Centric Learning from Videocited by 248
  3. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  4. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  5. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  6. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  7. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
  8. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  9. 2020Explainable Deep Learning: A Field Guide for the Uninitiatedcited by 127
  10. 2017Neural Expectation Maximizationcited by 123
  11. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  12. 2019Distilling Policy Distillationcited by 84
  13. 2021A Survey on Interpretable Reinforcement Learningcited by 73
  14. 2016Tagger: Deep Unsupervised Perceptual Groupingcited by 69
  15. 2019Neural Logic Reinforcement Learningcited by 53
  16. 2023An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learningcited by 49
  17. 2023Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstractioncited by 46
  18. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  19. 2022A Comparative Study of Faithfulness Metrics for Model Interpretability Methodscited by 39
  20. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  21. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  22. 2023Learning to Intervene on Concept Bottleneckscited by 29
  23. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  24. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  25. 2024Pix2Code: Learning to Compose Neural Visual Concepts as Programscited by 18
  26. 2022Boosting Object Representation Learning via Motion and Object Continuitycited by 17
  27. 2023Learning by Self-Explainingcited by 17
  28. 2024Neural Concept Bindercited by 14
  29. 2024BlendRL: A Framework for Merging Symbolic and Neural Policy Learningcited by 14
  30. 2022MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learningcited by 12
  31. 2021Efficient Decompositional Rule Extraction for Deep Neural Networkscited by 11
  32. 2022Explainability in reinforcement learning: perspective and positioncited by 11
  33. 2024Right on Time: Revising Time Series Models by Constraining their Explanationscited by 7
  34. 2022Image Segmentation-based Unsupervised Multiple Objects Discoverycited by 4
  35. 2023One Explanation Does Not Fit XILcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  2. 2025Deep Reinforcement Learning via Object-Centric Attentioncited by 2
  3. 2026Boosting deep Reinforcement Learning using pretraining with Logical Optionscited by 0
  4. 2026GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
  2. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networks
  3. 2026Sample-Efficient Neurosymbolic Deep Reinforcement Learning
  4. 2025Neural DNF-MT: A Neuro-symbolic Approach for Learning Interpretable and Editable Policies
  5. 2025Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning
  6. 2025BASIL: Best-Action Symbolic Interpretable Learning for Evolving Compact RL Policies
  7. 2025SymDQN: Symbolic Knowledge and Reasoning in Neural Network-based Reinforcement Learning
  8. 2024Neural Reward Machines