# CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs

arXiv: https://arxiv.org/abs/2511.14937 · HTML: https://mlsanity.com/paper/2511.14937/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2025.

## Built on by (most cited first, 26 of 26)

- [A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle](https://mlsanity.com/paper/2604.16548/) (2026)
- [Hidden in Memory: Sleeper Memory Poisoning in LLM Agents](https://mlsanity.com/paper/2605.15338/) (2026)
- [CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents](https://mlsanity.com/paper/2604.21308/) (2026)
- [GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents](https://mlsanity.com/paper/2606.18829/) (2026)
- [Contextualized Privacy Defense for LLM Agents](https://mlsanity.com/paper/2603.02983/) (2026)
- [AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks](https://mlsanity.com/paper/2604.01487/) (2026)
- [MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents](https://mlsanity.com/paper/2605.09530/) (2026)
- [Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models](https://mlsanity.com/paper/2601.15220/) (2026)
- [Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents](https://mlsanity.com/paper/2606.30306/) (2026)
- [Personalizing Agent Privacy Decisions via Logical Entailment](https://mlsanity.com/paper/2512.05065/) (2025)
- [From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents](https://mlsanity.com/paper/2603.18382/) (2026)
- [Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?](https://mlsanity.com/paper/2606.23189/) (2026)
- [Mitigating Over-Personalization in LLMs via Structured Memory](https://mlsanity.com/paper/2608.08300/) (2026)
- [PrivacyAlign: Contextual Privacy Alignment for LLM Agents](https://mlsanity.com/paper/2606.21710/) (2026)
- [PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems](https://mlsanity.com/paper/2607.05318/) (2026)
- [Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems](https://mlsanity.com/paper/2605.27766/) (2026)
- [IDP-Bench: Benchmarking ability of LLMs to protect personal information in interdependent privacy contexts](https://mlsanity.com/paper/2606.09908/) (2026)
- [Authorization Before Context: A Model-Neutral Audience Boundary Against Cross-Audience Memory Leakage in Agentic Systems](https://mlsanity.com/paper/2608.17148/) (2026)
- [Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection](https://mlsanity.com/paper/2609.04533/) (2026)
- [Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control](https://mlsanity.com/paper/2609.14003/) (2026)
- [Compositional Policy Violations: When Step-Level Compliance Fails In Agentic AI Workflows](https://mlsanity.com/paper/2609.18820/) (2026)
- [PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use](https://mlsanity.com/paper/2609.34526/) (2026)
- [EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents](https://mlsanity.com/paper/2609.35233/) (2026)
- [Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It](https://mlsanity.com/paper/2610.04794/) (2026)
- [Loose Lips: Understanding Privacy Leakage in Agentic Dialogue](https://mlsanity.com/paper/2610.14071/) (2026)
- [PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?](https://mlsanity.com/paper/2610.19314/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5's violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this - models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling.