# FORTIS: Benchmarking Over-Privilege in Agent Skills

arXiv: https://arxiv.org/abs/2605.09163 · HTML: https://mlsanity.com/paper/2605.09163/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 9 May 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 30 of 37)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [VPGTrans: Transfer Visual Prompt Generator across LLMs](https://mlsanity.com/paper/2305.01278/) (2023)
- [Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework](https://mlsanity.com/paper/2601.11893/) (2026)
- [Testing Language Model Agents Safely in the Wild](https://mlsanity.com/paper/2311.10538/) (2023)
- [Panoptic Scene Graph Generation with Semantics-Prototype Learning](https://mlsanity.com/paper/2307.15567/) (2023)
- [Treble Counterfactual VLMs: A Causal Approach to Hallucination](https://mlsanity.com/paper/2503.06169/) (2025)
- [CoAct-1: Computer-using Multi-Agent System with Coding Actions](https://mlsanity.com/paper/2508.03923/) (2025)
- [On the Importance of Building High-quality Training Datasets for Neural Code Search](https://mlsanity.com/paper/2202.06649/) (2022)
- [DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection](https://mlsanity.com/paper/2411.08227/) (2024)
- [AD-LLM: Benchmarking Large Language Models for Anomaly Detection](https://mlsanity.com/paper/2412.11142/) (2024)
- [DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning](https://mlsanity.com/paper/2503.11892/) (2025)
- [Secure On-Device Video OOD Detection Without Backpropagation](https://mlsanity.com/paper/2503.06166/) (2025)
- [Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Tooling](https://mlsanity.com/paper/2602.09084/) (2026)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [PyOD 2: A Python Library for Outlier Detection with LLM-powered Model Selection](https://mlsanity.com/paper/2412.12154/) (2024)
- [Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning](https://mlsanity.com/paper/2504.06438/) (2025)
- [Mitigating Hallucinations in Large Language Models via Causal Reasoning](https://mlsanity.com/paper/2508.12495/) (2025)

## Built on by (most cited first, 13 of 13)

- [RaMem: Contextual Reinstatement for Long-term Agentic Memory](https://mlsanity.com/paper/2606.22844/) (2026)
- [CatchBench: When Can an Agent Failure Be Caught?](https://mlsanity.com/paper/2608.22808/) (2026)
- [ResidualAuth: What Authorization State Must Language Agents Preserve under Revocable Delegation?](https://mlsanity.com/paper/2609.08062/) (2026)
- [JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles](https://mlsanity.com/paper/2607.27670/) (2026)
- [From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents](https://mlsanity.com/paper/2608.02171/) (2026)
- [OmniScientist: An Omni-Modal Omni-Discipline AI Scientist](https://mlsanity.com/paper/2608.13558/) (2026)
- [SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.11345/) (2026)
- [Agent Safety Is Action Alignment](https://mlsanity.com/paper/2606.28739/) (2026)
- [Task-Conditioned Least-Privilege Learning for Executable Terminal and MCP Agents](https://mlsanity.com/paper/2608.18351/) (2026)
- [Ajar: Measuring Open Privilege in Agent Defenses](https://mlsanity.com/paper/2609.26900/) (2026)
- [Controlled Decoding Attacks on Black-Box LLMs](https://mlsanity.com/paper/2609.36956/) (2026)
- [Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems](https://mlsanity.com/paper/2610.01042/) (2026)
- [Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding](https://mlsanity.com/paper/2610.07342/) (2026)

## Abstract (arXiv)

Large language model agents increasingly operate through an intermediate skill layer that mediates between user intent and concrete task execution. This layer is widely treated as an organizational abstraction, but we argue it is also a privilege boundary that current models routinely exceed. We present FORTIS, a benchmark that evaluates over-privilege in agent skills across two stages: whether a model selects the minimally sufficient skill from a large overlapping library, and whether it executes that skill without expanding into broader tools or actions than the skill permits. Across ten frontier models and three domains, we find that over-privileged behavior is the norm rather than the exception. Models consistently reach for higher-privilege skills and tools than the task requires, failing at both stages at rates that remain high even for the strongest available models. Failure is especially severe under the ordinary conditions of real user interaction: incomplete specification, convenience framing, and proximity to skill boundaries. None of these requires adversarial construction. The results indicate that the skill layer, far from containing agent behavior, is itself a primary source of privilege escalation in current systems.