mlsanityData to 14 Oct 2026

Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner

arXiv 2504.08247 ↗First posted 11 Apr 2025cs.LG cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

State-based sequence models like RWKV-7 offer a compelling alternative to Transformer architectures, achieving linear complexity while demonstrating greater expressive power in short-context scenarios and enabling state tracking beyond the (TC0) complexity class. However, RWKV-7 lacks mechanisms for token-parameter interactions and native scalability, limiting its adaptability and growth without retraining. In this paper, we propose Meta-State, a novel extension to RWKV-7 that replaces attention mechanisms with a fully state-driven approach, integrating token-parameter interactions through a Self-State Encoder (SSE) mechanism. The SSE repurposes a portion of the RWKV-7 Weighted Key-Value (WKV) state as transformation weights to encode token-parameter interactions in a linear, state-driven manner without introducing new trainable matrices or softmax operations, while preserving the autoregressive property of token processing. Meta-State supports progressive model scaling by expanding the WKV state and parameter tokens, reusing existing parameters without retraining. Our approach bridges the gap between state-based modeling, token-parameter interactions, and scalable architectures, offering a flexible framework for efficient and adaptable sequence modeling with linear complexity and constant memory usage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 13 of its 18 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

13 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  9. 2015Generating Sentences from a Continuous Spacecited by 612
  10. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  11. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  12. 2024TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameterscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024S7: Selective and Simplified State Space Layers for Sequence Modeling
  2. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
  3. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
  4. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  5. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
  6. 2024Attamba: Attending To Multi-Token States
  7. 2026State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking
  8. 2024Longhorn: State Space Models are Amortized Online Learners