# When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

arXiv: https://arxiv.org/abs/2602.14161 · HTML: https://mlsanity.com/paper/2602.14161/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 15 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)
- Mechanistic interpretability / [Steering and probing for LLM safety](https://mlsanity.com/mechinterp/steering-and-probing-for-llm-safety/)

## Builds on (most cited first, 30 of 30)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [LlamaFirewall: An open source guardrail system for building secure AI agents](https://mlsanity.com/paper/2505.03574/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [PromptArmor: Simple yet Effective Prompt Injection Defenses](https://mlsanity.com/paper/2507.15219/) (2025)
- [Get my drift? Catching LLM Task Drift with Activation Deltas](https://mlsanity.com/paper/2406.00799/) (2024)
- [PromptShield: Deployable Detection for Prompt Injection Attacks](https://mlsanity.com/paper/2501.15145/) (2025)
- [Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks](https://mlsanity.com/paper/2601.04603/) (2026)
- [Sparse Autoencoder Features for Classifications and Transferability](https://mlsanity.com/paper/2502.11367/) (2025)
- [LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge](https://mlsanity.com/paper/2506.09956/) (2025)
- [Large Language Models Encode Semantics and Alignment in Linearly Separable Representations](https://mlsanity.com/paper/2507.09709/) (2025)
- [AlignSentinel: Alignment-Aware Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2602.13597/) (2026)
- [Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders](https://mlsanity.com/paper/2506.23951/) (2025)
- [AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2602.13547/) (2026)

## Built on by (most cited first, 11 of 11)

- [Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift](https://mlsanity.com/paper/2606.22659/) (2026)
- [When Local Monitors Miss Compositional Harm: Diagnosing Distributed Backdoors in Multi-Agent Systems](https://mlsanity.com/paper/2607.11751/) (2026)
- [The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators](https://mlsanity.com/paper/2607.13075/) (2026)
- [Latent Undertow: How Ordinary Typos Break Probes](https://mlsanity.com/paper/2609.15994/) (2026)
- [Geometry-Lite: Interpretable Safety Probing via Layer-Wise Margin Geometry](https://mlsanity.com/paper/2605.20241/) (2026)
- [Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals](https://mlsanity.com/paper/2605.26999/) (2026)
- [Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks](https://mlsanity.com/paper/2608.09624/) (2026)
- [Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations](https://mlsanity.com/paper/2608.16970/) (2026)
- [Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild](https://mlsanity.com/paper/2610.02413/) (2026)
- [Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents](https://mlsanity.com/paper/2610.03448/) (2026)
- [Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection](https://mlsanity.com/paper/2610.19452/) (2026)

## Abstract (arXiv)

Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization. We train activation-based classifiers (linear probes on LLM hidden states) on a benchmark of 18 datasets (prompt attacks plus benign sources) and propose Leave-One-Dataset-Out (LODO) evaluation, where the held-out dataset is never seen during training. Across four LLMs from three families (Llama-3.1-8B, Gemma-3-27B, Qwen-3.5-2B/4B), standard cross-validation reports a pooled AUC 8.0-16.5 points higher than LODO; per-dataset held-out-test-vs-LODO accuracy gaps span 1-25 points. To understand the gap, we analyze the LODO stability of a linear probe's per-feature classifier coefficients, defining a retention metric for sparse-autoencoder (SAE) features that flags dataset-dependent shortcuts. 28-44% of top SAE features are shortcuts across models, a dataset-identity classifier reaches 96.6%, and the dataset-identifying and safety-relevant subspaces partially overlap. Standard domain-generalization fixes such as adversarial training, subspace projection, sample reweighting, and class balancing do not close the gap. Finally, we show LODO-weighted SAE attributions filter dataset artifacts for more reliable per-prompt explanations. We release our framework at https://github.com/maxf-zn/prompt-mining so future prompt-attack classifiers can be evaluated under LODO alongside CV.