mlsanityData to 14 Oct 2026

WuNeng: Hybrid State with Attention

arXiv 2504.19191 ↗First posted 27 Apr 2025cs.CLcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The WuNeng architecture introduces a novel approach to enhancing the expressivity and power of large language models by integrating recurrent neural network (RNN)-based RWKV-7 with advanced attention mechanisms, prioritizing heightened contextual coherence over reducing KV cache size. Building upon the hybrid-head concept from Hymba, WuNeng augments standard multi-head attention with additional RWKV-7 state-driven heads, rather than replacing existing heads, to enrich the model's representational capacity. A cross-head interaction technique fosters dynamic synergy among standard, state-driven, and newly introduced middle heads, leveraging concatenation, additive modulation, and gated fusion for robust information integration. Furthermore, a multi-token state processing mechanism harnesses the continuous RWKV-7 state to capture intricate, sequence-wide dependencies, significantly boosting expressivity. Remarkably, these enhancements are achieved with minimal additional parameters, ensuring efficiency while empowering the model to excel in complex reasoning and sequence generation tasks. WuNeng sets a new standard for balancing expressivity and computational efficiency in modern neural architectures.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 17 references that are in the map. First posted on arXiv 27 Apr 2025.

Builds on

16 of 17 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  9. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  10. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  11. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  12. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
4 more
  1. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  2. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  3. 2025Multi-Token Attentioncited by 10
  4. 2025ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformercited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  2. 2026Rethinking the Role of Efficient Attention in Hybrid Architecturescited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
  2. 2026The Wiola Architecture for Efficient Small Language Models
  3. 2025Native Hybrid Attention for Efficient Sequence Modeling
  4. 2025Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner
  5. 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
  6. 2024Taipan: Efficient and Expressive State Space Language Models with Selective Attention
  7. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  8. 2026Explicit Multi-head Attention for Inter-head Interaction in Large Language Models