# SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models

arXiv: https://arxiv.org/abs/2610.17727 · HTML: https://mlsanity.com/paper/2610.17727/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 28 of 28)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits](https://mlsanity.com/paper/2504.03767/) (2025)
- [Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks](https://mlsanity.com/paper/2005.09535/) (2020)
- ["Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild](https://mlsanity.com/paper/2602.06547/) (2026)
- [Formal Analysis and Supply Chain Security for Agentic AI Skills](https://mlsanity.com/paper/2603.00195/) (2026)
- [SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems](https://mlsanity.com/paper/2604.06811/) (2026)
- [How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study](https://mlsanity.com/paper/2604.03070/) (2026)
- [SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces](https://mlsanity.com/paper/2605.12015/) (2026)
- [MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills](https://mlsanity.com/paper/2606.07131/) (2026)
- [AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills](https://mlsanity.com/paper/2605.13940/) (2026)
- [AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor](https://mlsanity.com/paper/2601.05752/) (2026)
- [Practical Automated Detection of Malicious npm Packages](https://mlsanity.com/paper/2202.13953/) (2022)
- [SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction](https://mlsanity.com/paper/2606.02540/) (2026)
- [MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection](https://mlsanity.com/paper/2608.19901/) (2026)
- [RH-Detect: A Unified Benchmark for Reward Hacking Detection](https://mlsanity.com/paper/2610.10947/) (2026)

## Abstract (arXiv)

Agent skills turn natural-language instructions into actions, creating a security challenge: legitimate automation and malicious behaviour can look deceptively similar. A practical screen must catch nearly every attack while keeping the review load manageable. We present SkillScreen, a compact decision model trained to score skill maliciousness directly from SKILL.md in a single forward pass. On MaliciousSkillBench, SkillScreen flags 20-25% of benign skills at 99% recall on unseen attack families, substantially fewer than the word-level classifier and prompted LLM judges tested. Under the same training protocol, it also generalizes better than a fully fine-tuned encoder to unseen sources. Yet unfamiliar benign sources sharply increase false positives, showing that recognizing new attacks does not ensure reliable screening of new skill collections. Our results demonstrate the value of task-trained decision models for high-recall skill screening. Project page: https://jerroldh.github.io/SkillScreen-webpage/