# Mechanistic Anomaly Detection for "Quirky" Language Models

arXiv: https://arxiv.org/abs/2504.08812 · HTML: https://mlsanity.com/paper/2504.08812/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [SPECTRE: Defending Against Backdoor Attacks Using Robust Statistics](https://mlsanity.com/paper/2104.11315/) (2021)
- [Sycophancy in Large Language Models: Causes and Mitigations](https://mlsanity.com/paper/2411.15287/) (2024)
- [Eliciting Latent Knowledge from Quirky Language Models](https://mlsanity.com/paper/2312.01037/) (2023)
- [ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms](https://mlsanity.com/paper/2302.11408/) (2023)
- [Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations](https://mlsanity.com/paper/2311.09763/) (2023)
- [Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges](https://mlsanity.com/paper/2409.19993/) (2024)
- [Deep Probabilistic Models to Detect Data Poisoning Attacks](https://mlsanity.com/paper/1912.01206/) (2019)
- [Benchmarks for Detecting Measurement Tampering](https://mlsanity.com/paper/2308.15605/) (2023)

## Built on by (most cited first, 5 of 5)

- [Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation](https://mlsanity.com/paper/2606.11270/) (2026)
- [MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs](https://mlsanity.com/paper/2602.18782/) (2026)
- [Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies](https://mlsanity.com/paper/2511.03095/) (2025)
- [Interactions Between Crosscoder Features: A Compact Proofs Perspective](https://mlsanity.com/paper/2606.09940/) (2026)
- [AI Models Can Provably Hide Arbitrary Capabilities](https://mlsanity.com/paper/2610.13928/) (2026)

## Abstract (arXiv)

As LLMs grow in capability, the task of supervising LLMs becomes more challenging. Supervision failures can occur if LLMs are sensitive to factors that supervisors are unaware of. We investigate Mechanistic Anomaly Detection (MAD) as a technique to augment supervision of capable models; we use internal model features to identify anomalous training signals so they can be investigated or discarded. We train detectors to flag points from the test environment that differ substantially from the training environment, and experiment with a large variety of detector features and scoring rules to detect anomalies in a set of ``quirky'' language models. We find that detectors can achieve high discrimination on some tasks, but no detector is effective across all models and tasks. MAD techniques may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings.