# Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search

arXiv: https://arxiv.org/abs/2212.14849 · HTML: https://mlsanity.com/paper/2212.14849/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 30 Dec 2022.

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning](https://mlsanity.com/paper/1905.06088/) (2019)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [Explainability in Deep Reinforcement Learning](https://mlsanity.com/paper/2008.06693/) (2020)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Towards Deep Symbolic Reinforcement Learning](https://mlsanity.com/paper/1609.05518/) (2016)
- [Neuro-Symbolic Reinforcement Learning with First-Order Logic](https://mlsanity.com/paper/2110.10963/) (2021)
- [State of the Art Control of Atari Games Using Shallow Reinforcement Learning](https://mlsanity.com/paper/1512.01563/) (2015)
- [Symbolic Learning to Optimize: Towards Interpretability and Scalability](https://mlsanity.com/paper/2203.06578/) (2022)
- [Towards Symbolic Reinforcement Learning with Common Sense](https://mlsanity.com/paper/1804.08597/) (2018)
- [Symbolic Regression Methods for Reinforcement Learning](https://mlsanity.com/paper/1903.09688/) (2019)
- [Planning from Pixels in Atari with Learned Symbolic Representations](https://mlsanity.com/paper/2012.09126/) (2020)
- [Learning Symbolic Rules for Interpretable Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08228/) (2021)
- [Evolving simple programs for playing Atari games](https://mlsanity.com/paper/1806.05695/) (2018)
- [Deep Policies for Width-Based Planning in Pixel Domains](https://mlsanity.com/paper/1904.07091/) (2019)
- [Planning with Pixels in (Almost) Real Time](https://mlsanity.com/paper/1801.03354/) (2018)

## Built on by (most cited first, 9 of 9)

- [Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction](https://mlsanity.com/paper/2306.01439/) (2023)
- [Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation](https://mlsanity.com/paper/2305.00909/) (2023)
- [On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability](https://mlsanity.com/paper/2409.19924/) (2024)
- [End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations](https://mlsanity.com/paper/2403.12451/) (2024)
- [Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence](https://mlsanity.com/paper/2402.02701/) (2024)
- [EXPIL: Explanatory Predicate Invention for Learning in Games](https://mlsanity.com/paper/2406.06107/) (2024)
- [MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitation](https://mlsanity.com/paper/2602.05048/) (2026)
- [MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systems](https://mlsanity.com/paper/2501.19318/) (2025)
- [Generalizing the SINDy approach with nested neural networks](https://mlsanity.com/paper/2404.15742/) (2024)

## Abstract (arXiv)

Learning efficient and interpretable policies has been a challenging task in reinforcement learning (RL), particularly in the visual RL setting with complex scenes. While neural networks have achieved competitive performance, the resulting policies are often over-parameterized black boxes that are difficult to interpret and deploy efficiently. More recent symbolic RL frameworks have shown that high-level domain-specific programming logic can be designed to handle both policy learning and symbolic planning. However, these approaches rely on coded primitives with little feature learning, and when applied to high-dimensional visual scenes, they can suffer from scalability issues and perform poorly when images have complex object interactions. To address these challenges, we propose textitDifferentiable Symbolic Expression Search (DiffSES), a novel symbolic learning approach that discovers discrete symbolic policies using partially differentiable optimization. By using object-level abstractions instead of raw pixel-level inputs, DiffSES is able to leverage the simplicity and scalability advantages of symbolic expressions, while also incorporating the strengths of neural networks for feature learning and optimization. Our experiments demonstrate that DiffSES is able to generate symbolic policies that are simpler and more and scalable than state-of-the-art symbolic RL methods, with a reduced amount of symbolic prior knowledge.