Structure is not mechanism: high-gain gated-FFN rows across text and genomic foundation models
Not on the map. Why?
Abstract (arXiv)
A small number of unusually high-gain parameters can exert disproportionate effects in transformer language models, but whether analogous structures recur in genomic foundation models and whether structural geometry determines functional importance remains unknown. We analyzed high-gain rows in gated feed-forward networks across text and genomic foundation models, including a frozen 22-model causal census. Computing an associated bilinear weight operator exactly, without a diagonal approximation, we tested whether structural extremeness is a transferable mechanism. Activation-derived candidates were functionally enriched relative to random and top-norm same-layer controls, yet neither spectral concentration nor operator magnitude predicted causal effect size, and these associations vanished within the endpoint-homogeneous text-decoder subset. A within-layer sweep of 36 rows in one genomic and one text decoder resolved this into two regimes: below the detector's acceptance threshold the ratio carried no positive information about causal damage, whereas above it the ratio ordered rows strongly but did not grade severity as a dose-response. The same sweep revealed a second individually catastrophic row invisible to a one-candidate-per-model census, and non-additive damage among co-located critical rows. Case studies showed divergent causal organizations: a robust super-additive pair interaction in DNABERT-2, and in GENERator a sharply position-localized dependence in which preserving or restoring the row's beginning-of-sequence contribution rescued essentially all native-loss damage. High-gain gated-FFN rows are therefore a recurrent architectural phenotype whose structural prominence acts as an enrichment signal, not a calibrated measure of functional criticality or a specification of causal organization. Enrichment is general, but the mechanism is model-specific.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 17 references that are in the map. First posted on arXiv 13 Sep 2026.
Builds on
9 of 17 references.
References · 9
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2024Massive Activations in Large Language Modelscited by 294
- 2024The Super Weight in Large Language Modelscited by 52
- 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
- 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
- 2026Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretabilitycited by 2
Built on by
Not cited yet.