# From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents

arXiv: https://arxiv.org/abs/2606.04990 · HTML: https://mlsanity.com/paper/2606.04990/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 67 of its 82 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 30 of 67)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)

## Built on by (most cited first, 30 of 37)

- [GRADE: Graph Representation of LLM Agent Dependency and Execution](https://mlsanity.com/paper/2606.22741/) (2026)
- [Local Epistemic Uncertainty Guided Active Sampling for Plug-and-play Diffusive Image Restoration](https://mlsanity.com/paper/2608.06981/) (2026)
- [When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory](https://mlsanity.com/paper/2608.25553/) (2026)
- [You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvement](https://mlsanity.com/paper/2608.07555/) (2026)
- [Open Problems in AI Incident Governance](https://mlsanity.com/paper/2607.05163/) (2026)
- [APPA: Recoverable Information-Flow Control for Real-World LLM Agents](https://mlsanity.com/paper/2607.24625/) (2026)
- [Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems](https://mlsanity.com/paper/2608.10216/) (2026)
- [Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective](https://mlsanity.com/paper/2608.30563/) (2026)
- [Toward Workflow-Aware Benchmarking for Healthcare NLP Agents](https://mlsanity.com/paper/2609.00296/) (2026)
- [A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memory](https://mlsanity.com/paper/2609.30813/) (2026)
- [Certifiable Semantic Agreement Among LLM Agents: What the Admissibility Instrument Decides](https://mlsanity.com/paper/2606.07316/) (2026)
- [Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech](https://mlsanity.com/paper/2607.05165/) (2026)
- [Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents](https://mlsanity.com/paper/2607.27484/) (2026)
- [Toward Personal Intelligence Through Cooperative Observation](https://mlsanity.com/paper/2608.17128/) (2026)
- [Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification](https://mlsanity.com/paper/2608.17247/) (2026)
- [ReguSim: Evaluating LLM Agent Rule Grounding in Financial Compliance](https://mlsanity.com/paper/2608.19974/) (2026)
- [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026)
- [Noise Floor Audit for Agent Benchmarks](https://mlsanity.com/paper/2608.22331/) (2026)
- [DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillation](https://mlsanity.com/paper/2608.22806/) (2026)
- [Toward Effective and Reliable LLM Agents via Dynamic Ontology](https://mlsanity.com/paper/2608.22974/) (2026)
- [From Traceability to Justifiability: Accountability Structures in Agentic Software Engineering](https://mlsanity.com/paper/2608.23610/) (2026)
- [Forward-Deployed Full-Stack Engineering for Autonomous Cloud MLOps](https://mlsanity.com/paper/2608.29615/) (2026)
- [Causal Attribution for Agentic Decisions: Estimators, Coupling, and a Traceability Specification](https://mlsanity.com/paper/2609.06445/) (2026)
- [MOLE: Detecting Insider Threats in AI Agents](https://mlsanity.com/paper/2609.06966/) (2026)
- [Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study](https://mlsanity.com/paper/2609.07395/) (2026)
- [Self-Cleaning and Captured Anyway: One Measured Primitive for Error in a Store an Agent Writes to Itself, and What a Falling Score Actually Measures](https://mlsanity.com/paper/2609.25052/) (2026)
- [LLM Agents Can Easily Tamper With Their Own Traces](https://mlsanity.com/paper/2609.30266/) (2026)
- [RoutePrism: Tracing Construction Order Effects in Agent Memory](https://mlsanity.com/paper/2609.34160/) (2026)
- [Janus: Evidence-Before-Effect Sagas and Offline-Verifiable Provenance for Agentic LLMs](https://mlsanity.com/paper/2609.38266/) (2026)
- [NAQD Env: A benchmark for selective withdrawal in language agents](https://mlsanity.com/paper/2609.38460/) (2026)

## Abstract (arXiv)

Large language model (LLM)-based agents are moving beyond text generation to planning, tool use, retrieval, memory access, environmental interaction, and multi-agent collaboration. Because their outputs depend on information and execution steps across an agent run, their execution processes are difficult to verify, debug, and audit. Even a correct final answer does not reveal which evidence supported its claims, whether tool calls were justified, how memory shaped decisions, or where execution failures originated. Relevant research addresses these questions across evidence attribution, tool-use safety, agent memory, and observability, but these areas emphasize different parts of the execution process. This survey brings them into a common framework through execution provenance, a typed graph of an agent run, and evidence tracing, its projection onto evidence-support relations, distinguishing evidence that supports a conclusion from execution that influences it. We introduce a six-dimensional taxonomy covering trace sources, evidence and execution units, provenance relations, tracing granularity and timing, representation forms, and trust functions, complemented by operational definitions of nine provenance relations and two cross-cutting attributes, provenance origin and trace assurance. Applying this framework to the surveyed systems and benchmarks reveals three gaps: limited integration of fine-grained provenance with multiple trust functions, no benchmark in our catalog that evaluates the complete provenance chain, and little reporting of tamper-evident or attested traces. These findings motivate research on unified trace schemas, semantic provenance, memory and multi-agent lineage, provenance-aware safety and recovery, realistic benchmarks, trace assurance, and privacy-aware auditing. Resources are available at https://github.com/xiaoqi-7/Agent-Tracing-Survey.