mlsanityData to 14 Oct 2026

DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPU

arXiv 2605.20936 ↗First posted 20 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Hybrid attention architectures are becoming an increasingly important paradigm for improving LLM inference efficiency while preserving model quality, making hybrid architecture design a central problem. Existing designs often rely on manual empirical rules or proxy-based selector signals for layer-wise operator allocation. Recent NAS-style systems such as Jet-Nemotron demonstrate the promise of automated hybrid architecture search. However, Jet-Nemotron's PostNAS search stages alone use 200B tokens, making such search pipelines difficult to use as routine methods for hybrid architecture design. We introduce DASH, a fast differentiable search framework for hybrid attention architecture design, which relaxes discrete layer-wise attention operator placement into continuous architecture logits, prepares reusable teacher-aligned linear candidates, and performs architecture-only search with model and operator weights frozen to significantly enhance search efficiency. On Qwen2.5-3B-Instruct, DASH consistently outperforms a comprehensive suite of existing selector-style hybrid attention design baselines, showing that direct differentiable search can discover stronger hybrid architectures. Moreover, DASH achieves stronger RULER performance than released Jet-Nemotron models while remaining competitive on overlapping short-context and general benchmarks. Notably, each DASH search run uses only 12.3M tokens and takes about 20 minutes on a single RTX Pro 6000 GPU, corresponding to merely 0.006% of the PostNAS search tokens reported by Jet-Nemotron. These results suggest that high-quality hybrid attention architectures can be obtained through minutes-level differentiable search, providing a promising direction for hybrid architecture design.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 33 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

27 of 33 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  10. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
15 more
  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  2. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  3. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  4. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 131
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  6. 2021Finetuning Pretrained Transformers into RNNscited by 96
  7. 2026Qwen3-Coder-Next Technical Reportcited by 71
  8. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 65
  9. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  10. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 39
  11. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24
  12. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  13. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  14. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  15. 2026Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Constructioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
  2. 2025Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
  3. 2025The Anatomy of a Triton Attention Kernel
  4. 2026Rethinking the Role of Efficient Attention in Hybrid Architectures
  5. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
  6. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
  7. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
  8. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference