How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth t grows jointly with width n. This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large t. We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth--width scaling regimes that govern signal propagation: (i) a subcritical regime t=o(√ n), in which the infinite-width approximation remains valid; (ii) a critical regime t~ c√ n, in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a supercritical regime tgg √ n, in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results show that what constitutes "large width" depends crucially on the architecture: recurrent weight sharing makes finite-width effects visible at substantially shorter depths than in feedforward networks.