mlsanityData to 7 Oct 2026

Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels

arXiv 2607.11897 ↗First posted 12 Jun 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Linear attention replaces softmax attention's growing KV cache with a fixed recurrent state, but this compression limits exact state tracking and long-context memory. We introduce Semidirect Fourier Delta Attention (SFDA), a phase-controlled generalization of Kimi Delta Attention that replaces real diagonal decay with block-rotational Fourier control: [ St=(I-βt ktkt^*)LambdatSt-1+βtktvt^*, qquad Lambdat=diag(αtodot eiθt). ] Our main result is a constructive chunk-WY factorization for products (At=Lambdat-utrt^*), giving [ Atcdots A1=Gammat-YtMtWt^* ] with rank growth bounded inside fixed chunks. This yields an exact affine chunk transfer, formal stability and complexity bounds, and a compact characterization of phase-plus-low-rank memory. We verify the algebra numerically and show in toy state-tracking experiments that SFDA learns cyclic memory where the phase-disabled KDA baseline remains near chance. Fused kernels and large-scale language-model comparisons are left to future work.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 35 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

16 of 35 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  7. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  8. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  9. 2022Transformers Learn Shortcuts to Automatacited by 260
  10. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  11. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  12. 2022Transformer Quality in Linear Timecited by 189
4 more
  1. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  2. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  3. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  4. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
  2. 2026OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
  3. 2026Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecture
  5. 2025On Structured State-Space Duality
  6. 2026DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding
  7. 2025Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
  8. 2026Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences