mlsanityData to 7 Oct 2026

Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders

arXiv 2605.16640 ↗First posted 15 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We investigate the expressive power of hybrid recurrent-attention decoders, a class of architectures used in recent open-source language models such as Qwen3-Next and its successors. These models combine Gated Attention heads with recurrent Gated DeltaNet heads. Is there a formal advantage, in terms of model expressivity or efficiency, to such a hybrid architecture? We show that there is. We define parity-conditioned retrieval task and show that under constant-precision assumption, a Qwen-style hybrid of Gated DeltaNet and Gated Attention solves this task with a constant scratchpad, or equivalently O(1) chain-of-thought steps. In contrast, no similar solution exists for pure Gated DeltaNet models, while pure Gated Attention requires at least a polynomial scratchpad.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

15 of 23 references.

References · 15

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  3. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  4. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  5. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  6. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  7. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  8. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
  9. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  10. 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 47
  11. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  12. 2023Logical Languages Accepted by Transformer Encoders with Hard Attentioncited by 37
3 more
  1. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  2. 2024Lower Bounds on the Expressivity of Recurrent Neural Language Modelscited by 4
  3. 2026Parity, Sensitivity, and Transformerscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
  2. 2026DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding
  3. 2024Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
  4. 2026A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention
  5. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
  6. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
  7. 2026dQwen3.5: Hybrid-Attention Diffusion Language Models
  8. 2026RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State