mlsanityData to 13 Oct 2026

Ultra-Sparse Memory Network

arXiv 2411.12364 ↗First posted 19 Nov 2024cs.LGcited by 19

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms MoE. In experiments, the largest UltraMem we train has 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget, paving the way for billions of slots or experts.

Key facts. Cited by 19 papers in the map (data through 13 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 39 of its 47 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

39 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mixture of A Million Experts4 Jul
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  6. 2016Layer Normalizationcited by 5,624
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,836
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
23 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,804
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  5. 2024Mixtral of Expertscited by 1,811
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,115
  10. 2020On Layer Normalization in the Transformer Architecturecited by 910
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 890
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 853
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 705
  15. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  16. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 402
  17. 2021Hash Layers For Large Sparse Modelscited by 237
  18. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 171
  19. 2019Large Memory Layers with Product Keyscited by 121
  20. 2023A Study on ReLU and Softmax in Transformercited by 53
  21. 2024LoTR: Low Tensor Rank Weight Adaptationcited by 29
  22. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  23. 2020Large Product Key Memory for Pretrained Language Modelscited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 11.

19 papers.

Sort:

By area:

  1. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 81
  2. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  3. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 21
  4. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 13
  5. 2026Fast-weight Product Key Memorycited by 12
  6. 2025Frac-Connections: Fractional Extension of Hyper-Connectionscited by 8
  7. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  8. 2026MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scalingcited by 5
  9. 2026Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Designcited by 5
  10. 2026Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memorycited by 5
  11. 2026L3: Large Lookup Layerscited by 2
  12. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
7 more
  1. 2025Mixture of Lookup Key-Value Expertscited by 1
  2. 2026Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Trainingcited by 1
  3. 2025State Tuning: State-based Test-Time Scaling on RWKV-7cited by 0
  4. 2025MIDUS: Memory-Infused Depth Up-Scalingcited by 0
  5. 2026SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networkscited by 0
  6. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  7. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  2. 2022Efficiently Scaling Transformer Inference
  3. 2025LLM Inference Acceleration via Efficient Operation Fusion
  4. 2026Mixture of Chapters: Scaling Learnt Memory in Transformers
  5. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
  6. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
  7. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
  8. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving