Mixture of Lookup Key-Value Experts
Not on the map. Why?
Abstract (arXiv)
Recent research has developed several LLM architectures suitable for inference on end-user devices, such as the Mixture of Lookup Experts (MoLE)~parencitejiemixture2025. A key feature of MoLE is that each token id is associated with a dedicated group of experts. For a given input, only the experts corresponding to the input token id will be activated. Since the communication overhead of loading this small number of activated experts into RAM during inference is negligible, expert parameters can be offloaded to storage, making MoLE suitable for resource-constrained devices. However, MoLE's context-independent expert selection mechanism, based solely on input ids, may limit model performance. To address this, we propose the Mixture of Lookup Key-Value Experts (MoLKV) model. In MoLKV, each expert is structured as a key-value pair. For a given input, the input-derived query interacts with the cached key-value experts from the current sequence, generating a context-aware expert output. This context-aware mechanism alleviates the limitation of MoLE, and experimental results demonstrate that MoLKV achieves significantly lower validation loss in small-scale evaluations.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 33 references that are in the map. First posted on arXiv 10 Dec 2025.
Builds on
31 of 33 references.
References · 31
- 2017Attention Is All You Needcited by 60,361
- 2025Qwen3 Technical Reportcited by 8,820
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2020GLU Variants Improve Transformercited by 1,804
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
19 more
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
- 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
- 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
- 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
- 2019Large Memory Layers with Product Keyscited by 124
- 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
- 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistancecited by 87
- 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
- 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 73
- 2024Mixture of A Million Expertscited by 72
- 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
- 2024Memory Layers at Scalecited by 48
- 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 45
- 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 44
- 2024Densing Law of LLMscited by 40
- 2025Scaling Embedding Layers in Language Modelscited by 20
- 2024Ultra-Sparse Memory Networkcited by 19
- 2025Mixture of Lookup Expertscited by 12
- 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.