mlsanityData to 14 Oct 2026

Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour

arXiv 2505.17801 ↗First posted 23 May 2025cs.AIcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Autonomous multi-agent systems (MAS) are useful for automating complex tasks but raise trust concerns due to risks such as miscoordination or goal misalignment. Explainability is vital for users' trust calibration, but explainable MAS face challenges due to complex environments, the human factor, and non-standardised evaluation. Leveraging the counterfactual effect size model and LLMs, we propose Agentic eXplanations via Interrogative Simulation (AXIS). AXIS generates human-centred action explanations for multi-agent policies by having an LLM interrogate an environment simulator using prompts like 'whatif' and 'remove' to observe and synthesise counterfactual information over multiple rounds. We evaluate AXIS on autonomous driving across ten scenarios for five LLMs with a comprehensive methodology combining robustness, subjective preference, correctness, and goal/action prediction with an external LLM as evaluator. Compared to baselines, AXIS improves perceived explanation correctness by at least 7.7% across all models and goal prediction accuracy by 23% for four models, with comparable action prediction accuracy, achieving the highest scores overall. Our code is open-sourced at https://github.com/gyevnarb/axis.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 45 of its 109 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

45 of 109 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  7. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
  8. 2018Deep learning for time series classification: a reviewcited by 576
  9. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  10. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  11. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  12. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
33 more
  1. 2025Multi-Agent Risks from Advanced AIcited by 163
  2. 2020Explainable Reinforcement Learning: A Surveycited by 141
  3. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 131
  4. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  5. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  6. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  7. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  8. 2023Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision supportcited by 66
  9. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  10. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 53
  11. 2021The Who in XAI: How AI Background Shapes Perceptions of AI Explanationscited by 48
  12. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 39
  13. 2020Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Mapscited by 30
  14. 2021Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Valuescited by 25
  15. 2021Long Document Summarization in a Low Resource Setting using Pretrained Language Modelscited by 24
  16. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  17. 2021Levels of explainable artificial intelligence for human-aligned conversational explanationscited by 21
  18. 2020TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environmentscited by 18
  19. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 13
  20. 2021Interpretable Goal Recognition in the Presence of Occluded Factors for Autonomous Vehiclescited by 12
  21. 2022Toward Policy Explanations for Multi-Agent Reinforcement Learningcited by 11
  22. 2023Bridging the Transparency Gap: What Can Explainable AI Learn From the AI Act?cited by 10
  23. 2023Causal Explanations for Sequential Decision-Making in Multi-Agent Systemscited by 10
  24. 2023Explainable Multi-Agent Reinforcement Learning for Temporal Queriescited by 9
  25. 2022A Meta Survey of Quality Evaluation Criteria in Explanation Methodscited by 8
  26. 2024Evaluating Explanations Through LLMs: Beyond Traditional User Studiescited by 8
  27. 2019Exploiting Causality for Selective Belief Filtering in Dynamic Bayesian Networks (Extended Abstract)cited by 7
  28. 2024Enabling MCTS Explainability for Sequential Planning Through Computation Tree Logiccited by 6
  29. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  30. 2018Recent Advances in Neural Program Synthesiscited by 5
  31. 2025Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learningcited by 4
  32. 2024People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AIcited by 2
  33. 2024Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimatorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraudcited by 2
  2. 2025Towards Ethical Multi-Agent Systems of Large Language Models: A Mechanistic Interpretability Perspectivecited by 1
  3. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Abstract Counterfactuals for Language Model Agents
  2. 2026Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
  3. 2022Decisions that Explain Themselves: A User-Centric Deep Reinforcement Learning Explanation System
  4. 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations
  5. 2026Should I Have Expressed a Different Intent? Counterfactual Generation for LLM-Based Autonomous Control
  6. 2026Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness
  7. 2026Explaining Reinforcement Learning Decisions in Self-adaptive Systems
  8. 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation