Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
Multi-turn prompt injection follows a known attack path -- trust-building, pivoting, escalation but text-level defenses miss covert attacks where individual turns appear benign. We show this attack path leaves an activation-level signature in the model's residual stream: each phase shift moves the activation, producing a total path length far exceeding benign conversations. We call this adversarial restlessness. Five scalar trajectory features capturing this signal lift conversation-level detection from 76.2% to 93.8% on synthetic held-out data. The signal replicates across four model families (24B-70B); probes are model-specific and do not transfer across architectures. Generalization is source-dependent: leave-one-source-out evaluation shows each of synthetic, LMSYS-Chat-1M, and SafeDialBench captures distinct attack distributions, with detection on real-world LMSYS reaching 47-71% when its distribution is represented in training. Combined three-source training achieves 89.4% detection at 2.4% false positive rate on a held-out mixed set. We further show that three-phase turn-level labels(benign/pivoting/adversarial) unique to our synthetic dataset are essential: binary conversation-level labels produce 50-59% false positives. These results establish adversarial restlessness as a reliable activation-level signal and characterize the data requirements for practical deployment.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 16 of its 24 references that are in the map. First posted on arXiv 30 Apr 2026.
Builds on
16 of 24 references.
Closest ancestors · 7
Same direction, newest first
- 2026DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMs18 Feb
- 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks16 Feb
- 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation11 Feb
- 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks2 Nov
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
- 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
- 2023Detecting Language Model Attacks with Perplexity27 Aug
Other references · 9
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
- 2024Programming Refusal with Conditional Activation Steeringcited by 191
- 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
- 2026Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarioscited by 13
Built on by
Not cited yet.