# Enabling MCTS Explainability for Sequential Planning Through Computation Tree Logic

arXiv: https://arxiv.org/abs/2407.10820 · HTML: https://mlsanity.com/paper/2407.10820/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [A Symbolic Approach to Explaining Bayesian Network Classifiers](https://mlsanity.com/paper/1805.03364/) (2018)
- [Towards Explainable Artificial Intelligence](https://mlsanity.com/paper/1909.12072/) (2019)
- [The Emerging Landscape of Explainable AI Planning and Decision Making](https://mlsanity.com/paper/2002.11697/) (2020)
- [Explainable Planning](https://mlsanity.com/paper/1709.10256/) (2017)
- [Neural Approximate Dynamic Programming for On-Demand Ride-Pooling](https://mlsanity.com/paper/1911.08842/) (2019)
- [Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations](https://mlsanity.com/paper/2002.01080/) (2020)
- [On the Computation of Necessary and Sufficient Explanations](https://mlsanity.com/paper/2203.10451/) (2022)
- [Algorithmic Concept-based Explainable Reasoning](https://mlsanity.com/paper/2107.07493/) (2021)
- [Toward Policy Explanations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.12568/) (2022)
- [An Online Decision-Theoretic Pipeline for Responder Dispatch](https://mlsanity.com/paper/1902.08274/) (2019)
- [An Online Approach to Solve the Dynamic Vehicle Routing Problem with Stochastic Trip Requests for Paratransit Services](https://mlsanity.com/paper/2203.15127/) (2022)

## Built on by (most cited first, 6 of 6)

- [RuAG: Learned-rule-augmented Generation for Large Language Models](https://mlsanity.com/paper/2411.03349/) (2024)
- [Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour](https://mlsanity.com/paper/2505.17801/) (2025)
- [Combining LLMs with Logic-Based Framework to Explain MCTS](https://mlsanity.com/paper/2505.00610/) (2025)
- [Towards Explaining Monte-Carlo Tree Search by Using Its Enhancements](https://mlsanity.com/paper/2506.13223/) (2025)
- [M2-PALE: A Framework for Explaining Multi-Agent MCTS--Minimax Hybrids via Process Mining and LLMs](https://mlsanity.com/paper/2604.14687/) (2026)
- [Toward Template-Free Explainability for Monte Carlo Tree Search](https://mlsanity.com/paper/2605.16524/) (2026)

## Abstract (arXiv)

Monte Carlo tree search (MCTS) is one of the most capable online search algorithms for sequential planning tasks, with significant applications in areas such as resource allocation and transit planning. Despite its strong performance in real-world deployment, the inherent complexity of MCTS makes it challenging to understand for users without technical background. This paper considers the use of MCTS in transportation routing services, where the algorithm is integrated to develop optimized route plans. These plans are required to meet a range of constraints and requirements simultaneously, further complicating the task of explaining the algorithm's operation in real-world contexts. To address this critical research gap, we introduce a novel computation tree logic-based explainer for MCTS. Our framework begins by taking user-defined requirements and translating them into rigorous logic specifications through the use of language templates. Then, our explainer incorporates a logic verification and quantitative evaluation module that validates the states and actions traversed by the MCTS algorithm. The outcomes of this analysis are then rendered into human-readable descriptive text using a second set of language templates. The user satisfaction of our approach was assessed through a survey with 82 participants. The results indicated that our explanatory approach significantly outperforms other baselines in user preference.