mlsanityData to 14 Oct 2026

Approximating Two-Layer Feedforward Networks for Efficient Transformers

arXiv 2310.10837 ↗First posted 16 Oct 2023cs.LG cs.NEcited by 27

Not on the map. Why?

Cited by 21 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How to reduce compute and memory requirements of neural networks (NNs) without sacrificing performance? Many recent works use sparse Mixtures of Experts (MoEs) to build resource-efficient large language models (LMs). Here we introduce several novel perspectives on MoEs, presenting a general framework that unifies various methods to approximate two-layer NNs (e.g., feedforward blocks of Transformers), including product-key memories (PKMs). Leveraging insights from this framework, we propose methods to improve both MoEs and PKMs. Unlike prior work that compares MoEs with dense baselines under the compute-equal condition, our evaluation condition is parameter-equal, which is crucial to properly evaluate LMs. We show that our MoEs are competitive with the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two different scales, while being much more resource efficient. This demonstrates that MoEs are relevant not only to extremely large LMs but also to any-scale resource-efficient LMs. Our code is public.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
24 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2020Rethinking Attention with Performerscited by 1,601
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2022Toy Models of Superpositioncited by 971
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2018Universal Transformerscited by 647
  11. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  13. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  14. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  15. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  16. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  17. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  18. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  19. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  20. 2019Large Memory Layers with Product Keyscited by 124
  21. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  22. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  23. 2023A Study on ReLU and Softmax in Transformercited by 53
  24. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 7; 2026: 7.

27 papers.

Sort:

By area:

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  2. 2024Mixture of A Million Expertscited by 72
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  4. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  5. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  6. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
  7. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 24
  8. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  9. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 21
  10. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 20
  11. 2024Ultra-Sparse Memory Networkcited by 19
  12. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 14
15 more
  1. 2026Fast-weight Product Key Memorycited by 13
  2. 2026Sparse Layers are Critical to Scaling Looped Language Modelscited by 11
  3. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
  4. 2024Learning Neural Networks with Sparse Activationscited by 6
  5. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  6. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  7. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  8. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 4
  9. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 4
  10. 2025Improving Routing in Sparse Mixture of Experts with Graph of Tokenscited by 4
  11. 2026MoRE: Mixture of Reused Expertscited by 3
  12. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  13. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0
  14. 2026Mixture of Predefined Experts: Maximizing Data Usage on Vertical Federated Learningcited by 0
  15. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
  2. 2022Efficient Sparsely Activated Transformers
  3. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
  4. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  7. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  8. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity