SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Delta-Rule recurrent models maintain a fixed-size state, enabling O(1) inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: localized norm explosion atop a relatively sparse substrate, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose State Anomaly Neutralization (SANE), which applies adaptive tanh compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range (3 ≤ α ≤ 5), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over 24,000×, SANE retains functional reasoning (33.46--35.56) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds (α ≥ 8) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.