# Rule-Guided Reinforcement Learning Policy Evaluation and Improvement

arXiv: https://arxiv.org/abs/2503.09270 · HTML: https://mlsanity.com/paper/2503.09270/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Neural Logic Machines](https://mlsanity.com/paper/1904.11694/) (2019)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [Neural Logic Reinforcement Learning](https://mlsanity.com/paper/1904.10729/) (2019)
- [Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction](https://mlsanity.com/paper/2306.01439/) (2023)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing](https://mlsanity.com/paper/1808.05353/) (2018)
- [TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environments](https://mlsanity.com/paper/2009.04743/) (2020)
- [Search-Based Testing of Reinforcement Learning](https://mlsanity.com/paper/2205.04887/) (2022)
- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.12449/) (2022)

## Built on by (most cited first, 4 of 4)

- [Explainably Safe Reinforcement Learning](https://mlsanity.com/paper/2606.04634/) (2026)
- [Phase-Aware Guidance Injection for Recurrent MAPPO in Assembly-Line Disruption Recovery](https://mlsanity.com/paper/2606.16330/) (2026)
- [ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies](https://mlsanity.com/paper/2607.07235/) (2026)
- [Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents](https://mlsanity.com/paper/2608.17524/) (2026)

## Abstract (arXiv)

We consider the challenging problem of using domain knowledge to improve deep reinforcement learning policies. To this end, we propose LEGIBLE, a novel approach, following a multi-step process, which starts by mining rules from a deep RL policy, constituting a partially symbolic representation. These rules describe which decisions the RL policy makes and which it avoids making. In the second step, we generalize the mined rules using domain knowledge expressed as metamorphic relations. We adapt these relations from software testing to RL to specify expected changes of actions in response to changes in observations. The third step is evaluating generalized rules to determine which generalizations improve performance when enforced. These improvements show weaknesses in the policy, where it has not learned the general rules and thus can be improved by rule guidance. LEGIBLE supported by metamorphic relations provides a principled way of expressing and enforcing domain knowledge about RL environments. We show the efficacy of our approach by demonstrating that it effectively finds weaknesses, accompanied by explanations of these weaknesses, in eleven RL environments and by showcasing that guiding policy execution with rules improves performance w.r.t. gained reward.