mlsanityData to 14 Oct 2026

Memory Augmented Language Models through Mixture of Word Experts

arXiv 2311.10768 ↗First posted 15 Nov 2023cs.CLcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Scaling up the number of parameters of language models has proven to be an effective approach to improve performance. For dense models, increasing model size proportionally increases the model's computation footprint. In this work, we seek to aggressively decouple learning capacity and FLOPs through Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary based routing functions and experts. Our proposed approach, dubbed Mixture of Word Experts (MoWE), can be seen as a memory augmented model, where a large set of word-specific experts play the role of a sparse memory. We demonstrate that MoWE performs significantly better than the T5 family of models with similar number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms regular MoE models on knowledge intensive tasks and has similar performance to more complex memory augmented approaches that often require to invoke custom mechanisms to search the sparse memory.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 32 of its 47 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

32 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
16 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  6. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  9. 2021Hash Layers For Large Sparse Modelscited by 237
  10. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  11. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  12. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  13. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  14. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  15. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 43
  16. 2022Knowledge Prompts: Injecting World Knowledge into Language Models through Soft Promptscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 0; 2026: 3.

9 papers.

  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  2. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  3. 2024Mixture of A Million Expertscited by 72
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  6. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 13
  7. 2026STEM: Scaling Transformers with Embedding Modulescited by 12
  8. 2026Adaptive Inverted-Index Routing for Granular Mixtures-of-Expertscited by 0
  9. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024A Survey on Mixture of Experts in Large Language Models
  3. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  5. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  6. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  7. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  8. 2023Adaptive Gating in Mixture-of-Experts based Language Models