Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations
On the map: Mechanistic interpretability → Interpreting biomedical and science models
Abstract (arXiv)
Language models encode task-relevant knowledge in internal representations that far exceeds their output performance, but whether mechanistic interpretability methods can bridge this knowledge-action gap has not been systematically tested. We compared four mechanistic interpretability methods -- concept bottleneck steering (Steerling-8B), sparse autoencoder feature steering, logit lens with activation patching, and linear probing with truthfulness separator vector steering (Qwen 2.5 7B Instruct) -- for correcting false-negative triage errors using 400 physician-adjudicated clinical vignettes (144 hazards, 256 benign). Linear probes discriminated hazardous from benign cases with 98.2% AUROC, yet the model's output sensitivity was only 45.1%, a 53-percentage-point knowledge-action gap. Concept bottleneck steering corrected 20% of missed hazards but disrupted 53% of correct detections, indistinguishable from random perturbation (p=0.84). SAE feature steering produced zero effect despite 3,695 significant features. TSV steering at high strength corrected 24% of missed hazards while disrupting 6% of correct detections, but left 76% of errors uncorrected. Current mechanistic interpretability methods cannot reliably translate internal knowledge into corrected outputs, with implications for AI safety frameworks that assume interpretability enables effective error correction.
Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Interpreting biomedical and science models, one of 13 directions in Mechanistic interpretability. Builds on 12 of its 31 references that are in the map. First posted on arXiv 18 Mar 2026.
Builds on
12 of 31 references.
References · 12
- 2022Large Language Models Encode Clinical Knowledgecited by 1,830
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2020Concept Bottleneck Modelscited by 1,032
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2023A Closer Look at the Intervention Procedure of Concept Bottleneck Modelscited by 64
- 2024Concept Bottleneck Large Language Modelscited by 43
- 2026Decomposing Physician Disagreement in HealthBenchcited by 1
Built on by
Citing papers per year: 2026: 17.
17 papers.
Sort:
By area:
- 2026Where Do Apparent LLM Clinical Triage Failures Arise? Localizing the Multiple-Choice Format Effectcited by 4
- 2026Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Modelscited by 3
- 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimescited by 2
- 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 2
- 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 1
- 2026Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Expresscited by 1
- 2026Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internalscited by 1
- 2026Legible Failures: Detecting and Repairing In-Context Binding Errorscited by 1
- 2026Generative Interpretability via Scalable Neuro-Symbolic Modelscited by 1
- 2026Readable Before Actionable: Causal Tracing of Indirect Prompt Injectioncited by 1
- 2026The Open-Box Fallacy: Why AI Deployment Needs a Calibrated Verification Regimecited by 0
- 2026Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Framescited by 0
5 more
- 2026Validation-Gated Causal Interventions for Interpreting High-Stakes Large Language Model Behavior: A Case Study in Suicidality Detectioncited by 0
- 2026Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memorycited by 0
- 2026How Do Language Models Choose Between Context and Memory?cited by 0
- 2026Where Does Jagged Competence Come From?cited by 0
- 2026Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgmentcited by 0
- 2026Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment6 Oct 2026
- 2026Readable Before Actionable: Causal Tracing of Indirect Prompt Injection4 Oct 2026
- 2026Where Does Jagged Competence Come From?2 Oct 2026
- 2026Generative Interpretability via Scalable Neuro-Symbolic Models11 Sep 2026
- 2026Legible Failures: Detecting and Repairing In-Context Binding Errors10 Sep 2026
- 2026How Do Language Models Choose Between Context and Memory?1 Sep 2026
- 2026Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory27 Jul 2026
- 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning30 Jun 2026
- 2026Validation-Gated Causal Interventions for Interpreting High-Stakes Large Language Model Behavior: A Case Study in Suicidality Detection19 Jun 2026
- 2026Where Do Apparent LLM Clinical Triage Failures Arise? Localizing the Multiple-Choice Format Effect28 May 2026
- 2026Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames28 May 2026
- 2026Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals25 May 2026
5 more
- 2026Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Express25 May 2026
- 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language15 May 2026
- 2026The Open-Box Fallacy: Why AI Deployment Needs a Calibrated Verification Regime11 May 2026
- 2026Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models8 May 2026
- 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes7 May 2026
- 2026Where Do Apparent LLM Clinical Triage Failures Arise? Localizing the Multiple-Choice Format Effectcited by 4
- 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimescited by 2
- 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 1
- 2026Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Expresscited by 1
- 2026Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internalscited by 1
- 2026Legible Failures: Detecting and Repairing In-Context Binding Errorscited by 1
- 2026The Open-Box Fallacy: Why AI Deployment Needs a Calibrated Verification Regimecited by 0
- 2026Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Framescited by 0
- 2026Validation-Gated Causal Interventions for Interpreting High-Stakes Large Language Model Behavior: A Case Study in Suicidality Detectioncited by 0
- 2026Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memorycited by 0
- 2026How Do Language Models Choose Between Context and Memory?cited by 0
- 2026Where Does Jagged Competence Come From?cited by 0
12 most cited of 13 All 13 (JSON)
- 2026Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment6 Oct 2026
- 2026Where Does Jagged Competence Come From?2 Oct 2026
- 2026Legible Failures: Detecting and Repairing In-Context Binding Errors10 Sep 2026
- 2026How Do Language Models Choose Between Context and Memory?1 Sep 2026
- 2026Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory27 Jul 2026
- 2026Validation-Gated Causal Interventions for Interpreting High-Stakes Large Language Model Behavior: A Case Study in Suicidality Detection19 Jun 2026
- 2026Where Do Apparent LLM Clinical Triage Failures Arise? Localizing the Multiple-Choice Format Effect28 May 2026
- 2026Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames28 May 2026
- 2026Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals25 May 2026
- 2026Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Express25 May 2026
- 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language15 May 2026
- 2026The Open-Box Fallacy: Why AI Deployment Needs a Calibrated Verification Regime11 May 2026
12 newest of 13 All 13 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models
- 2024Towards Unifying Interpretability and Control: Evaluation via Intervention
- 2024Mechanistic?
- 2026Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations
- 2026When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic Interpretability
- 2026Scaling Inherently Interpretable Language Models
- 2026Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewing
- 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models