# Textual Explanations and Their Evaluations for Reinforcement Learning Policy

arXiv: https://arxiv.org/abs/2601.02514 · HTML: https://mlsanity.com/paper/2601.02514/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 12 of 12)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant](https://mlsanity.com/paper/2501.17546/) (2025)
- [What Makes a Good Explanation?: A Harmonized View of Properties of Explanations](https://mlsanity.com/paper/2211.05667/) (2022)
- [LLMs for XAI: Future Directions for Explaining Explanations](https://mlsanity.com/paper/2405.06064/) (2024)
- [HighwayLLM: Decision-Making and Navigation in Highway Driving with RL-Informed Language Model](https://mlsanity.com/paper/2405.13547/) (2024)
- [What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks](https://mlsanity.com/paper/2403.12730/) (2024)
- [InterpNET: Neural Introspection for Interpretable Deep Learning](https://mlsanity.com/paper/1710.09511/) (2017)
- [CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systems](https://mlsanity.com/paper/2406.08101/) (2024)
- [From Feature Importance to Natural Language Explanations Using LLMs with RAG](https://mlsanity.com/paper/2407.20990/) (2024)
- [CGXplain: Rule-Based Deep Neural Network Explanations Using Dual Linear Programs](https://mlsanity.com/paper/2304.05207/) (2023)

## Abstract (arXiv)

Understanding a Reinforcement Learning (RL) policy is crucial for ensuring that autonomous agents behave according to human expectations. This goal can be achieved using Explainable Reinforcement Learning (XRL) techniques. Although textual explanations are easily understood by humans, ensuring their correctness remains a challenge, and evaluations in state-of-the-art remain limited. We present a novel XRL framework for generating textual explanations, converting them into a set of transparent rules, improving their quality, and evaluating them. Expert's knowledge can be incorporated into this framework, and an automatic predicate generator is also proposed to determine the semantic information of a state. Textual explanations are generated using a Large Language Model (LLM) and a clustering technique to identify frequent conditions. These conditions are then converted into rules to evaluate their properties, fidelity, and performance in the deployed environment. Two refinement techniques are proposed to improve the quality of explanations and reduce conflicting information. Experiments were conducted in three open-source environments to enable reproducibility, and in a telecom use case to evaluate the industrial applicability of the proposed XRL framework. This framework addresses the limitations of an existing method, Autonomous Policy Explanation, and the generated transparent rules can achieve satisfactory performance on certain tasks. This framework also enables a systematic and quantitative evaluation of textual explanations, providing valuable insights for the XRL field.