mlsanityData to 14 Oct 2026

Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents

arXiv 2606.30383 ↗First posted 29 Jun 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

A rapidly growing class of LLM agents is multi-party: the agent acts for a principal (who briefs it, sends follow-ups, and receives results) while also conversing in a separate channel with a counterparty whose interests may diverge (negotiating with a vendor, screening inbound requests, or mediating between employees). Here "help whoever you are talking to" is the wrong objective. The agent must stay loyal to the principal it represents without over-refusing the principal's own cooperative asks. We study this multi-party loyalty problem and contribute a measurement instrument, two mechanisms, and a structural lesson. PrincipalBench is a 75-item multi-turn benchmark with leak probes, dual judges, and an integrity-audit gate. Across 13 frontier subjects it exposes a sharp split (<=20% vs. 53.6-75.3% harm) invisible to single-turn safety evaluations: a selective cluster that declines adversarial probes while still following the principal's legitimate requests, and an over-refusing cluster that refuses broadly. (M1) A prompt-time loyalty scaffold (a fixed system prompt of seven prioritized rules, open-coded from 50+ failure trajectories) holds Claude-Sonnet to 19.4% harm and all nine selective subjects to <=20%. (M2) A per-token-KL distillation recipe transfers a prompted Qwen3-32B teacher into 8B Qwen3 and Llama-3.1 students, the strongest open-weight recipe we measure. (Lesson) Both mechanisms only move along a common leak/over-refusal trade-off rather than crossing it: improving one axis costs the other, and the jointly favorable outcome stays out of reach.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

26 of 40 references.

References · 26

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
14 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  2. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  3. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  4. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  5. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  6. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  7. 2025Defeating Prompt Injections by Designcited by 297
  8. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  9. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  10. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  11. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  12. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 94
  13. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  14. 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluationcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SovereignNegotiation-Bench: Evaluating User-Owned Personal Agents In Delegated Bargaining Under Privacy, Consent, Evidence, And Institutional Pressurecited by 1
  2. 2026REDDIT: Forgetting-Resistant Correction of Timestamp Drift in ASR via Replay-Based Distribution Editingcited by 0
  3. 2026CallScreenBench: Benchmarking Small Language Models as Phone Secretariescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Voluntary Collusion with Secret Tools in Competing LLM Agents
  2. 2026Counterparty Modeling is Not Strategy: The Limits of LLM Negotiators
  3. 2026Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
  4. 2026Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation
  5. 2026Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent Systems
  6. 2026Used Car Salesbots? Honesty and Credulity of LLMs as Bargaining Agents under Partial Information
  7. 2026Delegation Without Trust: An Empirical Gap Analysis of Identity, Authorization, and Runtime Governance in Multi-Agent LLM Systems
  8. 2026Whom Do AI Agents Work For? Role Assignment Induces Sponsorship Bias in LLM Recommenders