# Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity

arXiv: https://arxiv.org/abs/2202.02886 · HTML: https://mlsanity.com/paper/2202.02886/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 6 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [Using Natural Language for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/1903.02020/) (2019)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications](https://mlsanity.com/paper/1807.06333/) (2018)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making](https://mlsanity.com/paper/1804.07779/) (2018)
- [Leveraging Human Guidance for Deep Reinforcement Learning Tasks](https://mlsanity.com/paper/1909.09906/) (2019)
- [Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation](https://mlsanity.com/paper/2006.14804/) (2020)
- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [GPT3-to-plan: Extracting plans from text using GPT-3](https://mlsanity.com/paper/2106.07131/) (2021)
- [Composable Planning with Attributes](https://mlsanity.com/paper/1803.00512/) (2018)
- [Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queries](https://mlsanity.com/paper/1802.01604/) (2018)
- [Extracting Action Sequences from Texts Based on Deep Reinforcement Learning](https://mlsanity.com/paper/1803.02632/) (2018)
- [Planning and Execution using Inaccurate Models with Provable Guarantees](https://mlsanity.com/paper/2003.04394/) (2020)
- [CMAX++ : Leveraging Experience in Planning and Execution using Inaccurate Models](https://mlsanity.com/paper/2009.09942/) (2020)

## Built on by (most cited first, 22 of 22)

- [Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning](https://mlsanity.com/paper/2305.14909/) (2023)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model](https://mlsanity.com/paper/2310.02054/) (2023)
- [LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation](https://mlsanity.com/paper/2210.12631/) (2022)
- [Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors](https://mlsanity.com/paper/2402.04210/) (2024)
- [NAVIX: Scaling MiniGrid Environments with JAX](https://mlsanity.com/paper/2407.19396/) (2024)
- [ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization](https://mlsanity.com/paper/2402.14528/) (2024)
- [Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2405.15194/) (2024)
- [Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences](https://mlsanity.com/paper/2210.15906/) (2022)
- [MAPLE: A Framework for Active Preference Learning Guided by Large Language Models](https://mlsanity.com/paper/2412.07207/) (2024)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion](https://mlsanity.com/paper/2210.15096/) (2022)
- [Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learning](https://mlsanity.com/paper/2503.21406/) (2025)
- [Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning](https://mlsanity.com/paper/2207.05018/) (2022)
- [Curiosity-Driven Imagination: Discovering Plan Operators and Learning Associated Policies for Open-World Adaptation](https://mlsanity.com/paper/2503.04931/) (2025)
- [Few-Shot Neuro-Symbolic Imitation Learning for Long-Horizon Planning and Acting](https://mlsanity.com/paper/2508.21501/) (2025)
- [Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines](https://mlsanity.com/paper/2311.15960/) (2023)
- [The Price Is Not Right: Neuro-Symbolic Methods Outperform VLAs on Structured Long-Horizon Manipulation Tasks with Significantly Lower Energy Consumption](https://mlsanity.com/paper/2602.19260/) (2026)
- [Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learning](https://mlsanity.com/paper/2603.11351/) (2026)
- [Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation](https://mlsanity.com/paper/2604.03759/) (2026)
- [Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2608.02993/) (2026)

## Abstract (arXiv)

Creating reinforcement learning (RL) agents that are capable of accepting and leveraging task-specific knowledge from humans has been long identified as a possible strategy for developing scalable approaches for solving long-horizon problems. While previous works have looked at the possibility of using symbolic models along with RL approaches, they tend to assume that the high-level action models are executable at low level and the fluents can exclusively characterize all desirable MDP states. Symbolic models of real world tasks are however often incomplete. To this end, we introduce Approximate Symbolic-Model Guided Reinforcement Learning, wherein we will formalize the relationship between the symbolic model and the underlying MDP that will allow us to characterize the incompleteness of the symbolic model. We will use these models to extract high-level landmarks that will be used to decompose the task. At the low level, we learn a set of diverse policies for each possible task subgoal identified by the landmark, which are then stitched together. We evaluate our system by testing on three different benchmark domains and show how even with incomplete symbolic model information, our approach is able to discover the task structure and efficiently guide the RL agent towards the goal.