Causal Structure is Inducible but Functionally Decoupled: The Routing/Readout Boundary of a Typed Mechanism Library
Not on the map. Why?
Abstract (arXiv)
When a language model answers an interventional question, the computation it must perform depends on the type of evidence the query requires. We report a decoupling in how a transformer organizes causal knowledge: slot-by-type structure induced by type-level supervision organizes routing, yet remains functionally decoupled from answer readout. We establish this with a typed mechanism library -- discrete mechanism slots partitioned by evidence type, auditable at the state level -- on a causal-world benchmark with exact interventional ground truth, under a frozen protocol, at two scales (22.6M and 125M). Four preregistered findings. (i) Origin. Slot-by-type organization is induced by type-level supervision: absent in architecturally identical unsupervised controls, not buyable by content-free gating labels, and statistically attributable to the supervision signal, replicating at 125M under a powered preregistered protocol (all nine cells passed). (ii) Boundary. The induced structure is a typed routing index with a sharp routing/readout boundary: slot codes scaffold routing but do not drive answer readout (|Deltahaty| ≤ 3.4×10-6, zero collateral, three seeds, stable across a 5.6x scale window) -- we therefore make no behavioral-editability claim. (iii) Cost. The structure is free: LM quality matches a parameter-matched monolith within 0.0082 nats. (iv) Trust. The library state is exactly local under edit and bit-exactly revertible -- 250 single-edit and 1,000 stacked reverts per seed, zero failures. We further find that the unsupervised null itself moves with scale, so comparisons reusing a null calibrated at one scale may be confounded at another. Every claim is tied to a preregistered, machine-checkable criterion archived before the data it governs; the full audit trail, including one criterion we failed and how the frozen protocol handled it, is released as an appendix.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 12 Aug 2026.
Builds on
19 of 28 references.
References · 19
- 2017Attention Is All You Needcited by 60,361
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
- 2022Mass-Editing Memory in a Transformercited by 1,001
- 2020Object-Centric Learning with Slot Attentioncited by 876
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2022Mixture-of-Experts with Expert Choice Routingcited by 623
- 2021Fast Model Editing at Scalecited by 498
7 more
- 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
- 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
- 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
- 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
- 2023Can Large Language Models Infer Causation from Correlation?cited by 146
- 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 107
- 2019The Scientific Method in the Science of Machine Learningcited by 15
Built on by
Not cited yet.