# Contextual Integrity in LLMs via Reasoning and Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.04245 · HTML: https://mlsanity.com/paper/2506.04245/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving](https://mlsanity.com/paper/2505.17685/) (2025)
- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [GuardReasoner: Towards Reasoning-based LLM Safeguards](https://mlsanity.com/paper/2501.18492/) (2025)
- [H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking](https://mlsanity.com/paper/2502.12893/) (2025)
- [OverThink: Slowdown Attacks on Reasoning LLMs](https://mlsanity.com/paper/2502.02542/) (2025)
- [A Survey on Post-training of Large Language Models](https://mlsanity.com/paper/2503.06072/) (2025)
- [CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data](https://mlsanity.com/paper/2409.13903/) (2024)
- [Firewalls to Secure Dynamic LLM Agentic Networks](https://mlsanity.com/paper/2502.01822/) (2025)
- [ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrails](https://mlsanity.com/paper/2502.13458/) (2025)
- [To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models](https://mlsanity.com/paper/2502.12202/) (2025)
- [Position: Contextual Integrity is Inadequately Applied to Language Models](https://mlsanity.com/paper/2501.19173/) (2025)
- [Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization](https://mlsanity.com/paper/2410.05255/) (2024)
- [Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing](https://mlsanity.com/paper/2502.07829/) (2025)
- [TDBench: A Benchmark for Top-Down Image Understanding with Reliability Analysis of Vision-Language Models](https://mlsanity.com/paper/2504.03748/) (2025)

## Built on by (most cited first, 30 of 31)

- [JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation](https://mlsanity.com/paper/2509.22548/) (2025)
- [ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations](https://mlsanity.com/paper/2511.05359/) (2025)
- [MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge](https://mlsanity.com/paper/2507.21183/) (2025)
- [CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents](https://mlsanity.com/paper/2604.21308/) (2026)
- [Privacy Reasoning in Ambiguous Contexts](https://mlsanity.com/paper/2506.12241/) (2025)
- [Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation](https://mlsanity.com/paper/2512.16310/) (2025)
- [AI Agents May Always Fall for Prompt Injections](https://mlsanity.com/paper/2605.17634/) (2026)
- [Recursive Self-Evolving Agents via Held-Out Selection](https://mlsanity.com/paper/2606.28374/) (2026)
- [Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy](https://mlsanity.com/paper/2603.15646/) (2026)
- [Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generation](https://mlsanity.com/paper/2509.02040/) (2025)
- [Chain-of-Sanitized-Thoughts: Plugging PII Leakage in CoT of Large Reasoning Models](https://mlsanity.com/paper/2601.05076/) (2026)
- [It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharing](https://mlsanity.com/paper/2509.21712/) (2025)
- [Mitigating Over-Personalization in LLMs via Structured Memory](https://mlsanity.com/paper/2608.08300/) (2026)
- [Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control](https://mlsanity.com/paper/2508.05342/) (2025)
- [Position: Capability Control Should be a Separate Goal From Alignment](https://mlsanity.com/paper/2602.05164/) (2026)
- [Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations](https://mlsanity.com/paper/2604.00209/) (2026)
- [Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization](https://mlsanity.com/paper/2606.13949/) (2026)
- [PrivacyAlign: Contextual Privacy Alignment for LLM Agents](https://mlsanity.com/paper/2606.21710/) (2026)
- [Towards Interpretable Adversarial Examples via Sparse Adversarial Attack](https://mlsanity.com/paper/2506.17250/) (2025)
- [Explainable reinforcement learning from human feedback to improve alignment](https://mlsanity.com/paper/2512.13837/) (2025)
- [Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?](https://mlsanity.com/paper/2602.05023/) (2026)
- [BiMind: A Dual-Head Reasoning Model with Attention-Geometry Adapter for Incorrect Information Detection](https://mlsanity.com/paper/2604.06022/) (2026)
- [It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs](https://mlsanity.com/paper/2605.20258/) (2026)
- [Quality analysis and evaluation prediction of RAG retrieval based on machine learning algorithms](https://mlsanity.com/paper/2511.19481/) (2025)
- [BadLLM-TG: A Backdoor Defender powered by LLM Trigger Generator](https://mlsanity.com/paper/2603.15692/) (2026)
- [Reinforcing privacy reasoning in LLMs via normative simulacra from fiction](https://mlsanity.com/paper/2604.20904/) (2026)
- [MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale](https://mlsanity.com/paper/2608.02613/) (2026)
- [CIDER: A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignment](https://mlsanity.com/paper/2608.09164/) (2026)
- [Rethinking Agent Security as a Networking Problem](https://mlsanity.com/paper/2608.12172/) (2026)
- [Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories](https://mlsanity.com/paper/2609.36414/) (2026)

## Abstract (arXiv)

As the era of autonomous agents making decisions on behalf of users unfolds, ensuring contextual integrity (CI) -- what is the appropriate information to share while carrying out a certain task -- becomes a central question to the field. We posit that CI demands a form of reasoning where the agent needs to reason about the context in which it is operating. To test this, we first prompt LLMs to reason explicitly about CI when deciding what information to disclose. We then extend this approach by developing a reinforcement learning (RL) framework that further instills in models the reasoning necessary to achieve CI. Using a synthetic, automatically created, dataset of only ~700 examples but with diverse contexts and information disclosure norms, we show that our method substantially reduces inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Importantly, improvements transfer from this synthetic dataset to established CI benchmarks such as PrivacyLens that has human annotations and evaluates privacy leakage of AI assistants in actions and tool calls. Our code is available at: https://github.com/EricGLan/CI-RL