mlsanityData to 14 Oct 2026

Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents

arXiv 2608.17524 ↗First posted 18 Aug 2026cs.LGcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

This preliminary paper outlines a planned evaluation benchmark for Explainable Reinforcement Learning (XRL) methods. Current evaluations rely on functionally-grounded metrics like faithfulness and compactness, and on human-grounded proxies like subjective ratings or prediction accuracy. We suggest evaluating XRL methods by how effectively their generated explanations help to diagnose and fix malfunctioning reinforcement learning (RL) agents. We propose EvalXRL, a benchmark in which a Large Language Model (LLM) coding agent uses different XRL methods to diagnose a held-out malfunction in an RL agent, and then repair it. Our proposed benchmark iterates across (environment × malfunction × XRL method) tuples and uses the reward signal of the RL agents to form a final score for each XRL method. The coding agent may use the method interactively: invoke the XRL method, process its output, form new hypotheses on what is broken, and invoke the method again with parameters adjusted for testing these hypotheses. This closed-loop structure may be described as a simplified version of the scientific method. Some XRL methods provide self-evaluations that follow this pattern; we propose the first head-to-head comparison of multiple XRL methods in closed-loop usage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 71 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

54 of 71 references.

References · 54

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  4. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  9. 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
  10. 2019Explainable Machine Learning in Deploymentcited by 211
  11. 2022Defining and Characterizing Reward Hackingcited by 203
  12. 2018Programmatically Interpretable Reinforcement Learningcited by 193
38 more
  1. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  2. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  3. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  4. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  5. 2020Debugging Tests for Model Explanationscited by 147
  6. 2017Visualizing and Understanding Atari Agentscited by 141
  7. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  8. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
  9. 2025Auditing language models for hidden objectivescited by 83
  10. 2023Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision supportcited by 66
  11. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 61
  12. 2022Rethinking Explainability as a Dialogue: A Practitioner's Perspectivecited by 52
  13. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  14. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  15. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 39
  16. 2026AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviorscited by 38
  17. 2022The Utility of Explainable AI in Ad Hoc Human-Machine Teamingcited by 30
  18. 2023Explaining Reinforcement Learning with Shapley Valuescited by 26
  19. 2019Explaining Reinforcement Learning to Mere Mortals: An Empirical Studycited by 21
  20. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  21. 2021Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approachcited by 18
  22. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 14
  23. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11
  24. 2025Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMscited by 11
  25. 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agentscited by 10
  26. 2021Interactive Explanations: Diagnosis and Repair of Reinforcement Learning Based Agent Behaviorscited by 8
  27. 2024Evaluating Explanations Through LLMs: Beyond Traditional User Studiescited by 8
  28. 2025KompeteAI: Accelerated Autonomous Multi-Agent System for End-to-End Pipeline Generation for Machine Learning Problemscited by 8
  29. 2022Explaining Reinforcement Learning Policies through Counterfactual Trajectoriescited by 7
  30. 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agentscited by 5
  31. 2018Towards a Grounded Dialog Model for Explainable Artificial Intelligencecited by 4
  32. 2023Mutation Testing of Deep Reinforcement Learning Based on Real Faultscited by 4
  33. 2025Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learningcited by 4
  34. 2025Rule-Guided Reinforcement Learning Policy Evaluation and Improvementcited by 4
  35. 2026Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?cited by 4
  36. 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviourcited by 3
  37. 2024XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniquescited by 2
  38. 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Textual Explanations and Their Evaluations for Reinforcement Learning Policy
  2. 2026Explaining Reinforcement Learning Agents via Inductive Logic Programming
  3. 2025A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs
  4. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models
  5. 2025Explainable Reinforcement Learning Agents Using World Models
  6. 2026BXRL: Behavior-Explainable Reinforcement Learning
  7. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
  8. 2025A Survey on Explainable Deep Reinforcement Learning