mlsanityData to 14 Oct 2026

Detecting High-Stakes Interactions with Activation Probes

arXiv 2506.10805 ↗First posted 12 Jun 2025cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Monitoring is an important aspect of safely deploying Large Language Models (LLMs). This paper examines activation probes for detecting ``high-stakes'' interactions -- where the text indicates that the interaction might lead to significant harm -- as a critical, yet underexplored, target for such monitoring. We evaluate several probe architectures trained on synthetic data, and find them to exhibit robust generalization to diverse, out-of-distribution, real-world data. Probes' performance is comparable to that of prompted or finetuned medium-sized LLM monitors, while offering computational savings of six orders-of-magnitude. These savings are enabled by reusing activations of the model that is being monitored. Our experiments also highlight the potential of building resource-aware hierarchical monitoring systems, where probes serve as an efficient initial filter and flag cases for more expensive downstream analysis. We release our novel synthetic dataset and the codebase at https://github.com/arrrlex/models-under-pressure.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

39 of 64 references.

References · 39

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
27 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 539
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  4. 2024Alignment faking in large language modelscited by 360
  5. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  6. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  7. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  8. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  9. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  10. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  11. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  12. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  13. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  14. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  15. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
  16. 2024Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classificationcited by 70
  17. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  18. 2025Detecting Strategic Deception Using Linear Probescited by 68
  19. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  20. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  21. 2024Obfuscated Activations Bypass LLM Latent-Space Defensescited by 45
  22. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 32
  23. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  24. 2023Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networkscited by 17
  25. 2025Investigating task-specific prompts and sparse autoencoders for activation monitoringcited by 15
  26. 2024A Taxonomy of Systemic Risks from General-Purpose AIcited by 12
  27. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 43.

53 papers.

Sort:

By area:

  1. 2026Building Production-Ready Probes For Geminicited by 47
  2. 2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 41
  3. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 40
  4. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 16
  5. 2026Diffusion Probe: Generated Image Result Prediction Using CNN Probescited by 14
  6. 2025Beyond Transcription: Mechanistic Interpretability in ASRcited by 13
  7. 2025Difficulties with Evaluating a Deception Detector for AIscited by 13
  8. 2025Combining Cost-Constrained Runtime Monitors for AI Safetycited by 10
  9. 2025The Impact of Off-Policy Training Data on Probe Generalisationcited by 9
  10. 2025Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitorscited by 9
  11. 2026Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failurescited by 8
  12. 2026Detecting Multi-Agent Collusion Through Multi-Agent Interpretabilitycited by 7
38 more
  1. 2026Verbalizing LLMs' assumptions to explain and control sycophancycited by 5
  2. 2026Monitoring the Internal Monologue: Probe Trajectories Reveal Reasoning Dynamicscited by 5
  3. 2025How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn Conversationscited by 4
  4. 2026Training Deliberative Monitors for Black-Box Scheming Detectioncited by 4
  5. 2025DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responsescited by 3
  6. 2026Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascadescited by 3
  7. 2026Segment-Level Coherence for Robust Harmful Intent Probing in LLMscited by 3
  8. 2026Beyond the Black Box: Interpretability of Agentic AI Tool Usecited by 3
  9. 2026Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agentscited by 3
  10. 2026AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agentcited by 3
  11. 2025Linear probes rely on textual evidence: Results from leakage mitigation studies in language modelscited by 2
  12. 2026Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalcited by 2
  13. 2026The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicatorscited by 2
  14. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationscited by 2
  15. 2025Red-teaming Activation Probes using Prompted LLMscited by 1
  16. 2026NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agentscited by 1
  17. 2026A Robust Evaluation of Probe Robustness: Lessons for Reliable OOD Uncertainty Quantificationcited by 1
  18. 2026Online Conformal Prediction Beyond Feedbackcited by 1
  19. 2026LLMs Learn to Evade Latent Monitors from Prior Feedback Alonecited by 1
  20. 2026Enabling Performant and Flexible Model-Internal Observability for LLM Inferencecited by 0
  21. 2026D2-Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation Signalscited by 0
  22. 2026Trait-space Monitoring for Emergent Misalignment During Supervised Finetuningcited by 0
  23. 2026Screening of Biosecurity Features in Metagenomic Data with Evo 2 Probescited by 0
  24. 2026Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?cited by 0
  25. 2026Activation Probes Surface Code-Security Signals that the Model's Output Missescited by 0
  26. 2026ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributionscited by 0
  27. 2026Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Biascited by 0
  28. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  29. 2026In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?cited by 0
  30. 2026Safety Monitors Mostly Catch What the Model Already Refusescited by 0
  31. 2026Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputscited by 0
  32. 2026Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?cited by 0
  33. 2026When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safetycited by 0
  34. 2026Removing the NEEDLE in the Haystack: Backdoor Removal in LLMs via Weight Orthogonalisationcited by 0
  35. 2026Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wildcited by 0
  36. 2026From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agentscited by 0
  37. 2026PyINE: A Framework for Scalable Elicitation and Oversight via Code Executioncited by 0
  38. 2026Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needscited by 0

50 most cited of 53 All 53 (JSON)