# Structural Inference: Interpreting Small Language Models with Susceptibilities

arXiv: https://arxiv.org/abs/2504.18274 · HTML: https://mlsanity.com/paper/2504.18274/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2025.

## Built on by (most cited first, 13 of 13)

- [Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units](https://mlsanity.com/paper/2601.21996/) (2026)
- [Patterning: The Dual of Interpretability](https://mlsanity.com/paper/2601.13548/) (2026)
- [From Global to Local: A Scalable Benchmark for Local Posterior Sampling](https://mlsanity.com/paper/2507.21449/) (2025)
- [Embryology of a Language Model](https://mlsanity.com/paper/2508.00331/) (2025)
- [Stagewise Reinforcement Learning and the Geometry of the Regret Landscape](https://mlsanity.com/paper/2601.07524/) (2026)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [From Memorization to Reasoning in the Spectrum of Loss Curvature](https://mlsanity.com/paper/2510.24256/) (2025)
- [Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning](https://mlsanity.com/paper/2605.07980/) (2026)
- [Mechanistic Anomaly Detection via Functional Attribution](https://mlsanity.com/paper/2604.18970/) (2026)
- [Linear Response Estimators for Singular Statistical Models](https://mlsanity.com/paper/2605.07970/) (2026)
- [Interpreting Reinforcement Learning Agents with Susceptibilities](https://mlsanity.com/paper/2605.08007/) (2026)
- [Dead Directions: Geometric Singular Learning](https://mlsanity.com/paper/2606.05957/) (2026)
- [Patterning in Practice: Debiasing Reward Models with Susceptibilities](https://mlsanity.com/paper/2609.00699/) (2026)

## Abstract (arXiv)

We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution, for example shifting the Pile toward GitHub or legal text, induces a first-order change in the posterior expectation of an observable localized on a chosen component of the network. The resulting susceptibility can be estimated efficiently with local SGLD samples and factorizes into signed, per-token contributions that serve as attribution scores. We combine these susceptibilities into a response matrix whose low-rank structure separates functional modules such as multigram and induction heads in a 3M-parameter transformer.