mlsanityData to 14 Oct 2026

Fast Feedforward Networks

arXiv 2308.14711 ↗First posted 28 Aug 2023cs.LG cs.AI cs.PFcited by 11

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

We break the linear link between the layer size and its inference cost by introducing the fast feedforward (FFF) architecture, a log-time alternative to feedforward networks. We demonstrate that FFFs are up to 220x faster than feedforward networks, up to 6x faster than mixture-of-experts networks, and exhibit better training properties than mixtures of experts thanks to noiseless conditional execution. Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 12 of its 18 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

12 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  8. 2020Rewriting a Deep Generative Modelcited by 107
  9. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  10. 2015Dynamic Capacity Networkscited by 16
  11. 2016Decision Forests, Convolutional Networks and the Models in-Betweencited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 5.

11 papers.

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  2. 2024Mixture of A Million Expertscited by 72
  3. 2025Position: A Theory of Deep Learning Must Include Compositional Sparsitycited by 14
  4. 2023Exponentially Faster Language Modellingcited by 4
  5. 2024Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layerscited by 2
  6. 2024Enhancing Fast Feed Forward Networks with Load Balancing and a Master Leaf Nodecited by 1
  7. 2026Dynamic sparsity in tree-structured feed-forward layers at scalecited by 0
  8. 2026LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deploymentcited by 0
  9. 2026HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networkscited by 0
  10. 2026Amplitude-Only FFN Intervention for Tool-Structured LLM Inference: Method, Gated Evaluation Protocol, and Cross-Model Empirical Resultscited by 0
  11. 2026BRANCH-MoE: Balance-Aware Tree Routing for Large Embedding Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparsely gated tiny linear experts
  2. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformers
  3. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
  4. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion
  5. 2026Improving MoE Compute Efficiency by Composing Weight and Data Sparsity
  6. 2023PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off
  7. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  8. 2026Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion