mlsanityData to 7 Oct 2026

Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks

arXiv 1312.4461 ↗First posted 16 Dec 2013cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Scalability properties of deep neural networks raise key research questions, particularly as the problems considered become larger and more challenging. This paper expands on the idea of conditional computation introduced by Bengio, et. al., where the nodes of a deep network are augmented by a set of gating units that determine when a node should be calculated. By factorizing the weight matrix into a low-rank approximation, an estimation of the sign of the pre-nonlinearity activation can be efficiently obtained. For networks using rectified-linear hidden units, this implies that the computation of a hidden unit with an estimated negative pre-nonlinearity can be ommitted altogether, as its value will become zero when nonlinearity is applied. For sparse neural networks, this can result in considerable speed gains. Experimental results using the MNIST and SVHN data sets with a fully-connected deep neural network demonstrate the performance robustness of the proposed scheme with respect to the error introduced by the conditional computation process.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 1; 2020: 1; 2021: 11; 2022: 6; 2023: 7; 2024: 9; 2025: 7; 2026: 1.

44 papers.

Sort:

By area:

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  4. 2021Dynamic Neural Networks: A Surveycited by 405
  5. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  6. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  7. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  8. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  9. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  10. 2024Mixture of A Million Expertscited by 70
  11. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  12. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
32 more
  1. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  2. 2021Zoo-Tuning: Adaptive Transfer from a Zoo of Modelscited by 35
  3. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  4. 2021Early Exiting with Ensemble Internal Classifierscited by 19
  5. 2022DeepPerform: An Efficient Approach for Performance Testing of Resource-Constrained Neural Networkscited by 16
  6. 2022Spatial Mixture-of-Expertscited by 14
  7. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 13
  8. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  9. 2023Fast Feedforward Networkscited by 10
  10. 2023Towards More Unified In-context Visual Understandingcited by 10
  11. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
  12. 2023DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimizationcited by 9
  13. 2024Intrinsic User-Centric Interpretability through Global Mixture of Expertscited by 9
  14. 2022Hub-Pathway: Transfer Learning from A Hub of Pre-trained Modelscited by 5
  15. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  16. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  17. 2022Understanding the Robustness of Multi-Exit Models under Common Corruptionscited by 4
  18. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  19. 2023Memorization Capacity of Neural Networks with Conditional Computationcited by 3
  20. 2019Learning Sparse Mixture of Experts for Visual Question Answeringcited by 2
  21. 2022Analysis of Branch Specialization and its Application in Image Decompositioncited by 2
  22. 2025Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsitycited by 2
  23. 2025The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Modelscited by 2
  24. 2021Dynamic Multi-Branch Layers for On-Device Neural Machine Translationcited by 1
  25. 2023Compression of Recurrent Neural Networks using Matrix Factorizationcited by 1
  26. 2024Video Relationship Detection Using Mixture of Expertscited by 1
  27. 2025Efficiency Robustness of Dynamic Deep Learning Systemscited by 1
  28. 2021CNN with large memory layerscited by 0
  29. 2021Taming the Beast: Learning to Control Neural Conversational Modelscited by 0
  30. 2023Deep Convolutional Tables: Deep Learning without Convolutionscited by 0
  31. 2025DynaNav: Dynamic Feature and Layer Selection for Efficient Visual Navigationcited by 0
  32. 2026Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exitcited by 0