mlsanityData to 7 Oct 2026

Hash Layers For Large Sparse Models

arXiv 2106.04426 ↗First posted 8 Jun 2021cs.LG cs.CLcited by 236

Not on the map. Why?

Cited by 173 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence. We show that this procedure either outperforms or is competitive with learning-to-route mixture-of-expert methods such as Switch Transformers and BASE Layers, while requiring no routing parameters or extra terms in the objective function such as a load balancing loss, and no sophisticated assignment algorithm. We study the performance of different hashing techniques, hash sizes and input features, and show that balanced and random hashes focused on the most local features work best, compared to either learning clusters or using longer-range context. We show our approach works well both on large language modeling and dialogue tasks, and on downstream fine-tuning tasks.

Key facts. Cited by 236 papers in the map (data through 7 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

28 of 39 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  12. 2020Recipes for building an open-domain chatbotcited by 593
16 more
  1. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  3. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  4. 2020The Pushshift Reddit Datasetcited by 289
  5. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  6. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  7. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  8. 2018Training Millions of Personalized Dialogue Agentscited by 126
  9. 2019Large Memory Layers with Product Keyscited by 120
  10. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  11. 2016Efficient softmax approximation for GPUscited by 67
  12. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  13. 2019The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agentscited by 45
  14. 2018Learning Semantic Textual Similarity from Conversationscited by 41
  15. 2016Generalizing and Hybridizing Count-based and Neural Language Modelscited by 10
  16. 2018Lightweight Adaptive Mixture of Neural and N-gram Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 34; 2023: 49; 2024: 60; 2025: 42; 2026: 43.

236 papers.

Sort:

By area:

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  6. 2023From Sparse to Soft Mixtures of Expertscited by 245
  7. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  9. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  10. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  11. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
38 more
  1. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  2. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  3. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  4. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  6. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  7. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  8. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  9. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  10. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  11. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  12. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  13. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  14. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  15. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  16. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  17. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  18. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  19. 2023Modular Deep Learningcited by 82
  20. 2024Mixture of A Million Expertscited by 70
  21. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  22. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  23. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  24. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  25. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  26. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
  27. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  28. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  29. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  30. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  31. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  32. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  33. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  34. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  35. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  36. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  37. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
  38. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46

50 most cited of 236 All 236 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026L3: Large Lookup Layers
  2. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformers
  3. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  4. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
  5. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  6. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
  7. 2026Sparse Layers are Critical to Scaling Looped Language Models
  8. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing