Linear Separability of Activation Representations after Supervised Fine-Tuning on Incorrect Responses: A Study of Synthetic Dishonesty in Large Language Models
On the map: Mechanistic interpretability → Probing deception, bias and beliefs
Abstract (arXiv)
When a language model is fine-tuned to produce systematically incorrect responses, does this training leave a structured, linearly recoverable trace in its internal activations? We study this question in a controlled model-organism setting using five transformer architectures spanning 1.4 to 9 billion parameters. For each model, an "honest" and a "dishonest" LoRA fine-tuned variant are constructed using identical question distributions but correct versus plausible-but-incorrect answers. Linear probes reach near-ceiling separability (AUC >= 0.9997) within the first few layers in four of five architectures. This separability transfers from TruthfulQA to held-out MMLU subjects for four models, but substantially less so for Pythia-1.4B. Six geometric analyses reveal an architectural dichotomy in representation structure. A control experiment further shows that LoRA fine-tuning itself leaves a strong activation fingerprint that is largely dissociable from the honesty-related signal, independent of fine-tuning data content. Two supplementary experiments qualify these findings: the identified direction transfers poorly to paraphrased questions, with AUC approaching chance, and norm-calibrated activation steering provides no reliable evidence of a causal role in generation, partly because stronger interventions disrupt output fluency. These results characterize what supervised fine-tuning on incorrect responses induces in activation space while clarifying important limitations on interpretation and causal significance.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability. Builds on 3 of its 3 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
3 of 3 references.
References · 3
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2021Fast Model Editing at Scalecited by 498
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models
- 2025When Truthful Representations Flip Under Deceptive Instructions?
- 2026Measuring Activation Control in Large Language Models
- 2026Closing the Confidence-Faithfulness Gap in Large Language Models
- 2026Diagnosing Correctness Probes under Self-Judgement Confounding
- 2026Grounding Without Corrective Control: Truth-Tracking Profiles for Large Language Models
- 2025Depth-Wise Activation Steering for Honest Language Models
- 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets