mlsanityData to 7 Oct 2026

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

arXiv 1308.3432 ↗First posted 15 Aug 2013cs.LGcited by 2,031

Not on the map. Why?

Cited by 69 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons? I.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and compare four families of solutions, applicable in different settings. One of them is the minimum variance unbiased gradient estimator for stochatic binary neurons (a special case of the REINFORCE algorithm). A second approach, introduced here, decomposes the operation of a binary stochastic neuron into a stochastic binary part and a smooth differentiable part, which approximates the expected effect of the pure stochatic binary neuron to first order. A third approach involves the injection of additive or multiplicative noise in a computational graph that is otherwise differentiable. A fourth approach heuristically copies the gradient with respect to the stochastic output directly as an estimator of the gradient with respect to the sigmoid argument (we call this the straight-through estimator). To explore a context where these estimators are useful, we consider a small-scale version of em conditional computation, where sparse stochastic units form a distributed representation of gaters that can turn off in combinatorially many ways large chunks of the computation performed in the rest of the neural network. In this case, it is important that the gating units produce an actual 0 most of the time. The resulting sparsity can be potentially be exploited to greatly reduce the computational cost of large deep networks for which conditional computation would be useful.

Key facts. Cited by 2,031 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Aug 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 1; 2020: 2; 2021: 266; 2022: 293; 2023: 341; 2024: 334; 2025: 376; 2026: 417.

2,031 papers.

Sort:

By area:

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  10. 2022High Fidelity Neural Audio Compressioncited by 673
  11. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  12. 2021A White Paper on Neural Network Quantizationcited by 453
38 more
  1. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  2. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  3. 2021Dynamic Neural Networks: A Surveycited by 405
  4. 2023Compact 3D Gaussian Representation for Radiance Fieldcited by 365
  5. 2022Transformers are Sample-Efficient World Modelscited by 317
  6. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  7. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  9. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 251
  10. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  11. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  12. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  13. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  14. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 220
  15. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  16. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  17. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  18. 2024HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compressioncited by 202
  19. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  20. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  21. 2021Block Pruning For Faster Transformerscited by 176
  22. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  23. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  24. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  25. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  26. 2022On-Device Training Under 256KB Memorycited by 158
  27. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  28. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  29. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  30. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  31. 2021Learned Token Pruning for Transformerscited by 138
  32. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  33. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  34. 2021Taming Visually Guided Sound Generationcited by 136
  35. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  36. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  37. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  38. 2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 127

50 most cited of 2,031 All 2,031 (JSON)