mlsanityData to 14 Oct 2026

Contextual Integrity in LLMs via Reasoning and Reinforcement Learning

arXiv 2506.04245 ↗First posted 29 May 2025cs.AI cs.CL cs.LGcited by 31

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

As the era of autonomous agents making decisions on behalf of users unfolds, ensuring contextual integrity (CI) -- what is the appropriate information to share while carrying out a certain task -- becomes a central question to the field. We posit that CI demands a form of reasoning where the agent needs to reason about the context in which it is operating. To test this, we first prompt LLMs to reason explicitly about CI when deciding what information to disclose. We then extend this approach by developing a reinforcement learning (RL) framework that further instills in models the reasoning necessary to achieve CI. Using a synthetic, automatically created, dataset of only ~700 examples but with diverse contexts and information disclosure norms, we show that our method substantially reduces inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Importantly, improvements transfer from this synthetic dataset to established CI benchmarks such as PrivacyLens that has human annotations and evaluates privacy leakage of AI assistants in actions and tool calls. Our code is available at: https://github.com/EricGLan/CI-RL

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 25 of its 49 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

25 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrails19 Feb
  2. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking18 Feb
  3. 2025To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models16 Feb
  4. 2025OverThink: Slowdown Attacks on Reasoning LLMs4 Feb
  5. 2025GuardReasoner: Towards Reasoning-based LLM Safeguards30 Jan
  6. 2024OpenAI o1 System Card21 Dec

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Mistral 7Bcited by 3,164
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  9. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  10. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  11. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  12. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
7 more
  1. 2025A Survey on Post-training of Large Language Modelscited by 54
  2. 2024CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Datacited by 37
  3. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 36
  4. 2025Position: Contextual Integrity is Inadequately Applied to Language Modelscited by 11
  5. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 9
  6. 2025Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editingcited by 6
  7. 2025TDBench: A Benchmark for Top-Down Image Understanding with Reliability Analysis of Vision-Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 20.

31 papers.

Sort:

By area:

  1. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  2. 2025ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversationscited by 16
  3. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 15
  4. 2026CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agentscited by 14
  5. 2025Privacy Reasoning in Ambiguous Contextscited by 11
  6. 2025Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigationcited by 11
  7. 2026AI Agents May Always Fall for Prompt Injectionscited by 9
  8. 2026Recursive Self-Evolving Agents via Held-Out Selectioncited by 7
  9. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  10. 2025Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generationcited by 5
  11. 2026Chain-of-Sanitized-Thoughts: Plugging PII Leakage in CoT of Large Reasoning Modelscited by 5
  12. 2025It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharingcited by 3
19 more
  1. 2026Mitigating Over-Personalization in LLMs via Structured Memorycited by 3
  2. 2025Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Controlcited by 2
  3. 2026Position: Capability Control Should be a Separate Goal From Alignmentcited by 2
  4. 2026Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representationscited by 2
  5. 2026Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitizationcited by 2
  6. 2026PrivacyAlign: Contextual Privacy Alignment for LLM Agentscited by 2
  7. 2025Towards Interpretable Adversarial Examples via Sparse Adversarial Attackcited by 1
  8. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  9. 2026Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?cited by 1
  10. 2026BiMind: A Dual-Head Reasoning Model with Attention-Geometry Adapter for Incorrect Information Detectioncited by 1
  11. 2026It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMscited by 1
  12. 2025Quality analysis and evaluation prediction of RAG retrieval based on machine learning algorithmscited by 0
  13. 2026BadLLM-TG: A Backdoor Defender powered by LLM Trigger Generatorcited by 0
  14. 2026Reinforcing privacy reasoning in LLMs via normative simulacra from fictioncited by 0
  15. 2026MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scalecited by 0
  16. 2026CIDER: A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignmentcited by 0
  17. 2026Rethinking Agent Security as a Networking Problemcited by 0
  18. 2026Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memoriescited by 0
  19. 2026AgentPrivArena: Evaluating and Auditing Real-world AI Agent Privacycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning
  2. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
  3. 2026Need to Know: Contextual-Integrity-Grounded Query Rewriting for Privacy-Conscious LLM Delegation
  4. 2026Contextualized Privacy Defense for LLM Agents
  5. 2026PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems
  6. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning
  7. 2026MPCI-Bench: A Benchmark for Multimodal Pairwise Contextual Integrity Evaluation of Language Model Agents
  8. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistants