# The Automaton Underneath: The Additive Input Pathway Is a Parasitic Attractor for State Tracking in Householder Linear RNN

arXiv: https://arxiv.org/abs/2609.18966 · HTML: https://mlsanity.com/paper/2609.18966/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 13 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024)
- [Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues](https://mlsanity.com/paper/2411.12537/) (2024)
- [DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products](https://mlsanity.com/paper/2502.10297/) (2025)
- [Understanding and Improving Length Generalization in Recurrent Models](https://mlsanity.com/paper/2507.02782/) (2025)
- [On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages](https://mlsanity.com/paper/2412.19350/) (2024)
- [The Expressive Limits of Diagonal SSMs for State-Tracking](https://mlsanity.com/paper/2603.01959/) (2026)
- [M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling](https://mlsanity.com/paper/2603.14360/) (2026)
- [Learning State-Tracking from Code Using Linear RNNs](https://mlsanity.com/paper/2602.14814/) (2026)
- [Revisiting Bi-Linear State Transitions in Recurrent Neural Networks](https://mlsanity.com/paper/2505.21749/) (2025)
- [Rethinking State Tracking in Recurrent Models Through Error Control Dynamics](https://mlsanity.com/paper/2605.07755/) (2026)
- [A Held-Out Transition-Pair Falsifier for Long-Horizon Non-Abelian State Tracking](https://mlsanity.com/paper/2606.07254/) (2026)
- [Robust Reasoning as a Symmetry-Protected Topological Phase](https://mlsanity.com/paper/2601.05240/) (2026)

## Abstract (arXiv)

Linear RNNs with input-dependent Householder-product transitions (DeltaNet/DeltaProduct-class) can provably represent hard state-tracking automata, yet trained models fail to length-generalize -- a gap recent work attributes to optimization, without a causal account. We give one, in a pre-registered, within-architecture causal ablation: the same model with one term deleted -- the additive input injection bt = Wb et. With bt, models fit length 32 and collapse out-of-distribution on parity, S4, A5, and non-solvable S5 word problems (0.20 at position 512 on S5). Without it -- input acting only through the orthogonal transitions -- the same architecture learns the exact automaton: median accuracy 1.00 at 16x the training length, at every width admitted by a representation law we state and test: the minimal number of Householder factors per token equals the maximal reflection length of the task's generators in the format-pinned representation (parity 1, S4 3, A5 and S5 4); below it, nothing fits. The contrast with DeltaProduct's S4/A5 at nh=2 (group-element classification, SO(3) realization) shows the law is representation-relative: task format is a hidden variable in state-tracking benchmarks. Two pre-registered arms locate the mechanism. (i) Initialized at a verified-exact solution with Wb=0, Adam grows the additive path and pulls the model off the exact solution; -b controls stay at 1.00. (ii) Our registered prediction that the fit routes through bt fired its kill criterion: all 49 fitting seeds retain in-domain fit under Wb:=0 -- and at law-minimal width, zeroing Wb at inference restores exact generalization (parity 5/5, S5 5/5, S4 4/5, A5 4/5). The additive pathway is parasitic: it destabilizes, then conceals, a correctly learned automaton. All 202 runs pre-registered; all numbers regenerate from artifacts.