mlsanityData to 7 Oct 2026

Red-teaming Activation Probes using Prompted LLMs

arXiv 2511.00554 ↗First posted 1 Nov 2025cs.LG cs.AIcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Activation probes are attractive monitors for AI systems due to low cost and latency, but their real-world robustness remains underexplored. We ask: What failure modes arise under realistic, black-box adversarial pressure, and how can we surface them with minimal effort? We present a lightweight black-box red-teaming procedure that wraps an off-the-shelf LLM with iterative feedback and in-context learning (ICL), and requires no fine-tuning, gradients, or architectural access. Running a case study with probes for high-stakes interactions, we show that our approach can help discover valuable insights about a SOTA probe. Our analysis uncovers interpretable brittleness patterns (e.g., legalese-induced FPs; bland procedural tone FNs) and reduced but persistent vulnerabilities under scenario-constraint attacks. These results suggest that simple prompted red-teaming scaffolding can anticipate failure patterns before deployment and might yield promising, actionable insights to harden future probes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 18 of its 21 references that are in the map. First posted on arXiv 1 Nov 2025.

Builds on

18 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration20 Mar
  2. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning24 Dec
  3. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Tester2 Oct
  4. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction25 Sep

Other references · 14

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  4. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  5. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  6. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  7. 2025Detecting Strategic Deception Using Linear Probescited by 65
  8. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  9. 2025Detecting High-Stakes Interactions with Activation Probescited by 45
  10. 2024Obfuscated Activations Bypass LLM Latent-Space Defensescited by 40
  11. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
  12. 2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 19
2 more
  1. 2025Investigating task-specific prompts and sparse autoencoders for activation monitoringcited by 13
  2. 2023Generating Valid and Natural Adversarial Examples with Large Language Modelscited by 12

Built on by

Not cited yet.