Faithfulness to Refusal: A Causal Audit of Neuron Selectors
Not on the map. Why?
Abstract (arXiv)
Attribution scores increasingly identify which neuron rows of a language model matter for applications such as pruning, interpretability, and editing for safety, yet whether they identify causally important rows is rarely tested directly. We address this with two paired audits built on one-shot neuron-row zeroing. We first audit selectors at the language-modeling level: attribution methods substantially outperform activation and magnitude-based baselines at identifying dispensable rows across five LLMs. We then adapt the same intervention into a behavior test by driving it with a contrastive harmful-versus-benign signal; the attributed rows are sufficient to install refusal on hate and crime while keeping benign over-refusal low and preserving language model fluency, and specific in that layer-matched random controls at the same depths fail. Highly rank-stable selectors can be among the least causally valid. Refusal moreover lives in a redundant subspace, where different attribution methods install it through largely disjoint row sets, so the recovered edit is one realization of a sufficient set rather than a unique mechanism. Together, these findings show that rank-stability proxies miss the kinds of selector failures a direct causal audit can surface.%
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 6 Jul 2026.
Builds on
33 of 40 references.
References · 33
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2017The Kinetics Human Action Video Datasetcited by 1,909
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
- 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
- 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,082
- 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
- 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 692
- 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
- 2022Kubric: A scalable dataset generatorcited by 381
- 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
21 more
- 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 207
- 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
- 2019Temporal Cycle-Consistency Learningcited by 154
- 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
- 2018Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networkscited by 133
- 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
- 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 95
- 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
- 2016Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networkscited by 73
- 2021CLDA: Contrastive Learning for Semi-Supervised Domain Adaptationcited by 52
- 2023RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptionscited by 49
- 2023STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimationcited by 37
- 2024StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videoscited by 37
- 2024StereoDiffusion: Training-Free Stereo Image Generation Using Latent Diffusion Modelscited by 25
- 2024SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrixcited by 20
- 2025ZeroStereo: Zero-shot Stereo Matching from Single Imagescited by 13
- 2025Depth Anything at Any Conditioncited by 12
- 2025Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversioncited by 10
- 2025M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversioncited by 10
- 2025Restereo: Diffusion stereo video generation and restorationcited by 7
- 2025DepthFocus: Controllable Depth Estimation for See-Through Scenescited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Targeted Neuron Modulation via Contrastive Pair Search
- 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
- 2026When Refusal Looks Safe: The Refusal-Cue Shortcut in Safety Guard Models
- 2025Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal
- 2026Tripwire: Triggering Aligned Refusal via Statistically Certified Safety Neurons
- 2026On the Steering Dimensionality of Refusal in Language Models
- 2026There Is More to Refusal in Large Language Models than a Single Direction
- 2024Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs