mlsanityData to 7 Oct 2026

Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety

arXiv 2506.05451 ↗First posted 5 Jun 2025cs.SE cs.AI cs.CLcited by 10

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

As large language models (LLMs) see wider real-world use, understanding and mitigating their unsafe behaviors is critical. Interpretation techniques can reveal causes of unsafe outputs and guide safety, but such connections with safety are often overlooked in prior surveys. We present the first survey that bridges this gap, introducing a unified framework that connects safety-focused interpretation methods, the safety enhancements they inform, and the tools that operationalize them. Our novel taxonomy, organized by LLM workflow stages, summarizes nearly 70 works at their intersections. We conclude with open challenges and future directions. This timely survey helps researchers and practitioners navigate key advancements for safer, more interpretable LLMs.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 255 of its 338 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

255 of 338 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety6 Mar
  2. 2025Language Models Can Predict Their Own Behavior18 Feb
  3. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment22 Jan
  4. 2025Tell me about yourself: LLMs are aware of their learned behaviors19 Jan
  5. 2024Large Language Model Safety: A Holistic Survey23 Dec
  6. 2024SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs11 Nov
  7. 2024On the Role of Attention Heads in Large Language Model Safety17 Oct
  8. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  9. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations17 Jun
  10. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  11. 2024Current state of LLM Risks and AI Guardrails16 Jun
  12. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
4 more
  1. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching22 May
  2. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vector18 Apr
  3. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safety1 Apr
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb

Other references · 239

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
38 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  3. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  11. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  12. 2022In-context Learning and Induction Headscited by 930
  13. 2022Toy Models of Superpositioncited by 928
  14. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  15. 2023Steering Language Models With Activation Engineeringcited by 824
  16. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  17. 2020Quantifying Attention Flow in Transformerscited by 759
  18. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  19. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  20. 2019Attention is not not Explanationcited by 629
  21. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  22. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  23. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  24. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  25. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  26. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  27. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  28. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  29. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  30. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  31. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  32. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  33. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  34. 2021Causal Abstractions of Neural Networkscited by 401
  35. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  36. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  37. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  38. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328

50 most cited of 239 All 239 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  2. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  3. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
  4. 2026All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMscited by 8
  5. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4
  6. 2026AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agentcited by 3
  7. 2026Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Modelscited by 1
  8. 2025InFerActive: Interactive Tree-Based Exploration of LLM Sampling for Safety Evaluationcited by 0
  9. 2026Attributing and Exploiting Safety Vectors through Global Optimization in Large Language Modelscited by 0
  10. 2026CR4T: Rewrite-Based Guardrails for Adolescent LLM Safetycited by 0