mlsanityData to 14 Oct 2026

Triggers and Diagnostics for LLM-Based Interpretability Failures in Active Inference Agents

arXiv 2609.23215 ↗First posted 19 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance also fits: General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM explainers are increasingly attached to autonomous agents as runtime oversight, with operators reading a generated account of the agent's beliefs and actions rather than its internal state. We audit the account itself, pairing an Active Inference (AIF) agent that tracks German grid demand and adjusts generation with an LLM explainer on three backends (GPT-4o, Claude-3-Opus, Gemini), and probing the pair with three black-box triggers. Corrupting the observation stream by 600 MW per step moves the agent's posterior by 490 MW, roughly 0.9% of grid capacity. None of the 30 explanations produced during the injection flag anything under a stated rubric, and each narrates the corrupted belief fluently. On timesteps where the agent takes an objectively wrong action, all three explainers produce a sycophantic rationalization 80-95% of the time (n = 20 per backend). Attacker-controlled text in the observation metadata field steers the explainer, with susceptibility differing by provider and data exfiltration succeeding on all three. We propose mitigations for each failure but do not evaluate them. In every failure we observed, the explanation was fluent and wrong. Moreover, nothing in the explainer architecture checks whether an explanation is true before an operator acts on it. Testing the explainer therefore belongs in any audit of an agentic deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 15 of its 25 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

15 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 14

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  7. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  9. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  10. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  11. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  12. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
2 more
  1. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  2. 2023Explaining black box text modules in natural language with language modelscited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Think Locally, Explain Globally: Graph-Guided LLM Investigations via Local Reasoning and Belief Propagation
  2. 2026Fail Loudly: An Auditable Runtime for Agentic Data Analysis
  3. 2026Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
  4. 2024Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents
  5. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection
  6. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
  7. 2026Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
  8. 2026Akashic: A Low-Overhead LLM Inference Service with MemAttention