mlsanityData to 14 Oct 2026

Block-Based Double Decoders

arXiv 2605.18807 ↗First posted 11 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Encoder-decoder models offer substantial inference-time savings over decoder-only models, but their pretraining objectives suffer from sparse supervision and dynamic sequence lengths, keeping them out of practice at scale. We propose block-based double decoders, a novel transformer architecture that utilizes doubly-causal block-based attention masks to train with full loss supervision and static sequence packing, combining decoder-only training efficiency with encoder-decoder inference efficiency. In scaling law experiments, block-based double decoders strongly outperform encoder-decoders and closely track decoder-only models across scales. At inference time, they cut KV-cache memory and per-token compute by at least 2/3 without sacrificing prefill caching or other existing inference optimizations available to decoder-only models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

15 of 18 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  11. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  12. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
3 more
  1. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  2. 2026Stability and Generalization in Looped Transformerscited by 10
  3. 2025Return of the Encoder: Maximizing Parameter Efficiency for SLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inference
  2. 2026End-to-End Context Compression at Scale
  3. 2025Efficient Pretraining Length Scaling
  4. 2025Training and Inference Efficiency of Encoder-Decoder Speech Models
  5. 2025Cross-Attention Speculative Decoding
  6. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models
  7. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
  8. 2024Tandem Transformers for Inference Efficient LLMs