mlsanityData to 14 Oct 2026

The Extender: A Log-Structured Transformer

arXiv 2609.32759 ↗First posted 26 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual h, a superposition channel. The Extender adds a concatenation channel x: each layer ℓ emits both a residual update δℓ which is added to h, and a much smaller extension εℓ which is appended to x. While both the FFN and q see h, the attention kv projections take only x as input. As a result, the fully extended x contains the complete input for the kv projections of all layers, reducing the persistent attention memory footprint from 2Ldmodel to sum|εℓ|. We find that with |εℓ|=32, the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters, and exceeds Transformer accuracy on long-context (RULER) workloads, again at 924M parameters. For our 1664-wide, 924M model, the Extender's persistent attention memory footprint is 104× smaller than MHA. The memory savings grow with model width.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

31 of 36 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
19 more
  1. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  9. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  10. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  11. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  12. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  13. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  14. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  15. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  16. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  17. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  18. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
  19. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressioncited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
  2. 2026Revisiting the Shape Convention of Transformer Language Models
  3. 2025Modality Agnostic Efficient Long Range Encoder
  4. 2026The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
  5. 2025Equivalence of Context and Parameter Updates in Modern Transformer Blocks
  6. 2024Theoretical limitations of multi-layer Transformer
  7. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformers
  8. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference