mlsanityData to 14 Oct 2026

PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines

arXiv 2605.10614 ↗First posted 11 May 2026cs.AIcited by 2

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Multi-agent LLM systems introduce a security risk in which sensitive information accessed by one agent can propagate through shared context and reappear in downstream outputs, even without explicit adversarial intent. We formalise this phenomenon as propagation amplification, where leakage risk increases across agent boundaries as sensitive content is repeatedly exposed to downstream generators. Existing defences, including prompt-based safeguards, static pattern matching, and LLM-as-judge filtering, are not designed for this setting: they either operate after generation, rely primarily on surface-form patterns, or add substantial latency without modelling the generation process itself. To resolve these issues, we propose PRISM, a real-time defence that treats credential leakage as a sequential risk accumulation problem during generation. At each decoding step, PRISM combines 16 signals spanning lexical, structural, information-theoretic, behavioural, and contextual features into a calibrated risk score, enabling per-token intervention through green, yellow, and red risk zones. Our central observation is that credential reproduction is often preceded by a measurable shift in generation dynamics, characterised by entropy collapse and increasing logit concentration. When combined with text-structural cues such as identifier-pattern detection, these temporal signals provide an early warning of leakage before a secret is fully reconstructed. Across a 2,000-task adversarial benchmark covering 13 attack categories and three pressure levels in a heterogeneous four-agent pipeline, PRISM achieves F1 = 0.832 with precision = 1.000 and recall = 0.712, while producing no observed leakage on our benchmark (0.0% task-level leak rate) and preserving output utility of 0.893. It substantially outperforms the strongest baseline, Span Tagger, which achieves F1 = 0.719 with a 15.0% task-level leak rate.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 24 of its 35 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LlamaFirewall: An open source guardrail system for building secure AI agents6 May

Other references · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  10. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  12. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
11 more
  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  2. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  3. 2023Privacy Issues in Large Language Models: A Surveycited by 60
  4. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
  5. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  6. 2025A Survey on Privacy Risks and Protection in Large Language Modelscited by 21
  7. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  8. 2024LLM Security Guard for Codecited by 15
  9. 2025ControlNET: A Firewall for RAG-based LLM Systemcited by 12
  10. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  11. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Trust propagation and structural containment in Multi-agent LLM pipelines
  2. 2026Observable Channels, Not Just Storage: Evaluating Privacy Leakage in LLM Agent Pipelines
  3. 2026When Agents Talk, Secrets Travel: Topology-Driven Leakage and Recipient-Conditioned Sanitization in Multi-Agent LLM Systems
  4. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach
  5. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems
  6. 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents
  7. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  8. 2026Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures