# Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

arXiv: https://arxiv.org/abs/2601.14004 · HTML: https://mlsanity.com/paper/2601.14004/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 307 of its 394 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 30 of 307)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)

## Built on by (most cited first, 27 of 27)

- [Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models](https://mlsanity.com/paper/2605.11887/) (2026)
- [Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation](https://mlsanity.com/paper/2604.10098/) (2026)
- [Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics](https://mlsanity.com/paper/2602.02343/) (2026)
- [Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance](https://mlsanity.com/paper/2602.01047/) (2026)
- [Emergent Analogical Reasoning in Transformers](https://mlsanity.com/paper/2602.01992/) (2026)
- [The Cylindrical Representation Hypothesis for Language Model Steering](https://mlsanity.com/paper/2605.01844/) (2026)
- [OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond](https://mlsanity.com/paper/2605.19660/) (2026)
- [SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining](https://mlsanity.com/paper/2602.10718/) (2026)
- [PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models](https://mlsanity.com/paper/2602.21428/) (2026)
- [Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training](https://mlsanity.com/paper/2605.06076/) (2026)
- [The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2605.26670/) (2026)
- [Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus](https://mlsanity.com/paper/2608.12149/) (2026)
- [DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning](https://mlsanity.com/paper/2606.00160/) (2026)
- [Reading the Gate, Not the Interference: Output-Side Interference Measurement Does Not Track Merge Collapse](https://mlsanity.com/paper/2608.11797/) (2026)
- [From Attribution to Action: A Human-Centered Application of Activation Steering](https://mlsanity.com/paper/2604.11467/) (2026)
- [Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training](https://mlsanity.com/paper/2605.11416/) (2026)
- [Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs](https://mlsanity.com/paper/2606.00726/) (2026)
- [Temporal Preference Concepts and their Functions in a Large Language Model](https://mlsanity.com/paper/2606.05194/) (2026)
- [Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms](https://mlsanity.com/paper/2606.08236/) (2026)
- [TIPS: Topological Ill-Posedness Probing and Steering in Large Language Models](https://mlsanity.com/paper/2606.23590/) (2026)
- [Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning](https://mlsanity.com/paper/2607.08393/) (2026)
- [CircuitKIT : Circuit Discovery, Evaluation, and Application Toolkit for Mechanistic Interpretability](https://mlsanity.com/paper/2607.19317/) (2026)
- [SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization](https://mlsanity.com/paper/2608.13538/) (2026)
- [Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning](https://mlsanity.com/paper/2608.24482/) (2026)
- [Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs](https://mlsanity.com/paper/2609.06473/) (2026)
- [Don't Judge an LLM Only by Its Activations: Discovering Suppressed Safety Features via Counterfactual Activation Potential](https://mlsanity.com/paper/2610.05541/) (2026)
- [Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs](https://mlsanity.com/paper/2610.07646/) (2026)

## Abstract (arXiv)

Mechanistic Interpretability (MI) has emerged as a vital approach to demystify the opaque decision-making of Large Language Models (LLMs). However, existing reviews primarily treat MI as an observational science, summarizing analytical insights while lacking a systematic framework for actionable intervention. To bridge this gap, we present a practical survey structured around the pipeline: "Locate, Steer, and Improve." We formally categorize Localizing (diagnosis) and Steering (intervention) methods based on specific Interpretable Objects to establish a rigorous intervention protocol. Furthermore, we demonstrate how this framework enables tangible improvements in Alignment, Capability, and Efficiency, effectively operationalizing MI as an actionable methodology for model optimization. The curated paper list of this work is available at https://github.com/rattlesnakey/Awesome-Actionable-MI-Survey.