mlsanityData to 14 Oct 2026

COSMIC: Generalized Refusal Direction Identification in LLM Activations

arXiv 2506.00085 ↗First posted 30 May 2025cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) encode behaviors such as refusal within their activation space, yet identifying these behaviors remains a significant challenge. Existing methods often rely on predefined refusal templates detectable in output tokens or require manual analysis. We introduce COSMIC (Cosine Similarity Metrics for Inversion of Concepts), an automated framework for direction selection that identifies viable steering directions and target layers using cosine similarity - entirely independent of model outputs. COSMIC achieves steering performance comparable to prior methods without requiring assumptions about a model's refusal behavior, such as the presence of specific refusal tokens. It reliably identifies refusal directions in adversarial settings and weakly aligned models, and is capable of steering such models toward safer behavior with minimal increase in false refusals, demonstrating robustness across a wide range of alignment conditions.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

37 of 59 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
25 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  4. 2021Ethical and social risks of harm from Language Modelscited by 966
  5. 2023Steering Language Models With Activation Engineeringcited by 922
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  9. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  10. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 344
  11. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  12. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  13. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  14. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  15. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  16. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  17. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  18. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  19. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 56
  20. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 40
  21. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 35
  22. 2024Refusal in LLMs is an Affine Functioncited by 29
  23. 2024Bilinear MLPs enable weight-based mechanistic interpretabilitycited by 28
  24. 2023A Geometric Notion of Causal Probingcited by 25
  25. 2022Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projectioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  2. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  3. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 12
  4. 2026Steered LLM Activations are Non-Surjectivecited by 8
  5. 2025ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behaviorcited by 7
  6. 2026Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometrycited by 2
  7. 2026Controlling Tool Use with Heading-Specific Activation Steeringcited by 2
  8. 2026Agent Security Needs Redefinition through a Holistic Frameworkcited by 2
  9. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  10. 2026RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMscited by 1
  11. 2026Ablating Safety: Mechanisms for Removing Alignment in Language Models for Security Applicationscited by 1
  12. 2026Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignmentcited by 0
7 more
  1. 2026Decomposing Refusal Steering in Mixture-of-Experts Modelscited by 0
  2. 2026From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuningcited by 0
  3. 2026Harnessing Textual Refusal Directions for Multimodal Safetycited by 0
  4. 2026Fast Multi-dimensional Refusal Subspaces via RFM-AGOPcited by 0
  5. 2026WildSEEK: Evaluating Language Models for Information-Seekingcited by 0
  6. 2026Component and Dimension Sparsity in Transformer Refusal Mechanismscited by 0
  7. 2026Narrative Paradoxes in LLM Inference: Shaping Reasoning Trajectories toward Misalignmentcited by 0