mlsanityData to 14 Oct 2026

Linearly Decoding Refused Knowledge in Aligned Language Models

arXiv 2507.00239 ↗First posted 30 Jun 2025cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 2 more
Abstract (arXiv)

Most commonly used language models (LMs) are instruction-tuned and aligned using a combination of fine-tuning and reinforcement learning, causing them to refuse users requests deemed harmful by the model. However, jailbreak prompts can often bypass these refusal mechanisms and elicit harmful responses. In this work, we study the extent to which information accessed via jailbreak prompts is decodable using linear probes trained on LM hidden states. We show that a great deal of initially refused information is linearly decodable. For example, across models, the response of a jailbroken LM for the average IQ of a country can be predicted by a linear probe with Pearson correlations exceeding 0.8. Surprisingly, we find that probes trained on base models (which do not refuse) sometimes transfer to their instruction-tuned versions and are capable of revealing information that jailbreaks decode generatively, suggesting that the internal representations of many refused properties persist from base LMs through instruction-tuning. Importantly, we show that this information is not merely "leftover" in instruction-tuned models, but is actively used by them: we find that probe-predicted values correlate with LM generated pairwise comparisons, indicating that the information decoded by our probes align with suppressed generative behavior that may be expressed more subtly in other downstream tasks. Overall, our results suggest that instruction-tuning does not wholly eliminate or even relocate harmful information in representation space-they merely suppress its direct expression, leaving it both linearly accessible and indirectly influential in downstream behavior.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 66 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

52 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models12 Jun
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  4. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  5. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
32 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  6. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  7. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  8. 2023Language Models Represent Space and Timecited by 316
  9. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  10. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  11. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  12. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  13. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  14. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  15. 2021Alignment of Language Agentscited by 168
  16. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  17. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  18. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 86
  19. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  20. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysiscited by 64
  21. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 59
  22. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 44
  23. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 40
  24. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 36
  25. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 32
  26. 2023Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibilitycited by 30
  27. 2024Do LLMs "know" internally when they follow instructions?cited by 26
  28. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  29. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 14
  30. 2024Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulationscited by 11
  31. 2025Language Models Can Predict Their Own Behaviorcited by 8
  32. 2021Probing Linguistic Information For Logical Inference In Pre-trained Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026HalluWorld: A Controlled Benchmark for Hallucination via Reference World Modelscited by 2
  2. 2026Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off
  2. 2026Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak
  3. 2025RAID: Refusal-Aware and Integrated Decoding for Jailbreaking LLMs
  4. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  5. 2026Activation Surgery: Jailbreaking White-box LLMs without Touching the Prompt
  6. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  7. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  8. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings