# Red-teaming Activation Probes using Prompted LLMs

arXiv: https://arxiv.org/abs/2511.00554 · HTML: https://mlsanity.com/paper/2511.00554/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 1 Nov 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 18 of 18)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends](https://mlsanity.com/paper/2407.07403/) (2024)
- [FLIRT: Feedback Loop In-context Red Teaming](https://mlsanity.com/paper/2308.04265/) (2023)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs](https://mlsanity.com/paper/2410.03768/) (2024)
- [Detecting High-Stakes Interactions with Activation Probes](https://mlsanity.com/paper/2506.10805/) (2025)
- [Automated Red Teaming with GOAT: the Generative Offensive Agent Tester](https://mlsanity.com/paper/2410.01606/) (2024)
- [Obfuscated Activations Bypass LLM Latent-Space Defenses](https://mlsanity.com/paper/2412.09565/) (2024)
- [AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration](https://mlsanity.com/paper/2503.15754/) (2025)
- [Probing and Steering Evaluation Awareness of Language Models](https://mlsanity.com/paper/2507.01786/) (2025)
- [Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction](https://mlsanity.com/paper/2409.16783/) (2024)
- [Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning](https://mlsanity.com/paper/2412.18693/) (2024)
- [RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?](https://mlsanity.com/paper/2506.14261/) (2025)
- [Investigating task-specific prompts and sparse autoencoders for activation monitoring](https://mlsanity.com/paper/2504.20271/) (2025)
- [Generating Valid and Natural Adversarial Examples with Large Language Models](https://mlsanity.com/paper/2311.11861/) (2023)

## Built on by (most cited first, 1 of 1)

- [Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs](https://mlsanity.com/paper/2610.10594/) (2026)

## Abstract (arXiv)

Activation probes are attractive monitors for AI systems due to low cost and latency, but their real-world robustness remains underexplored. We ask: What failure modes arise under realistic, black-box adversarial pressure, and how can we surface them with minimal effort? We present a lightweight black-box red-teaming procedure that wraps an off-the-shelf LLM with iterative feedback and in-context learning (ICL), and requires no fine-tuning, gradients, or architectural access. Running a case study with probes for high-stakes interactions, we show that our approach can help discover valuable insights about a SOTA probe. Our analysis uncovers interpretable brittleness patterns (e.g., legalese-induced FPs; bland procedural tone FNs) and reduced but persistent vulnerabilities under scenario-constraint attacks. These results suggest that simple prompted red-teaming scaffolding can anticipate failure patterns before deployment and might yield promising, actionable insights to harden future probes.