mlsanityData to 14 Oct 2026

Stochastic activations

arXiv 2509.22358 ↗First posted 26 Sep 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw. This strategy circumvents the optimization problem associated with RELU, namely, the constant shape for negative inputs that prevents the gradient flow. We leverage this strategy in two ways: (1) We use stochastic activations during pre-training and fine-tune the model with RELU, which is used at inference time to provide sparse latent vectors. This reduces the inference FLOPs and translates into a significant speedup on CPU and GPU. This leads to better results than training from scratch with the RELU activation function. (2) We evaluate stochastic activations for sequence generation. This strategy performs reasonably well: it has higher diversity and has only slightly inferior performance to the best deterministic non-linearity, SILU, combined with temperature sampling. This provides an alternative way to increase the diversity of generated text.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
26 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  14. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  15. 2017Self-Normalizing Neural Networkscited by 598
  16. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  17. 2018DropBlock: A regularization method for convolutional networkscited by 262
  18. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  19. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  20. 2019Large Memory Layers with Product Keyscited by 124
  21. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  22. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  23. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 59
  24. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  25. 2022Stochastic Adaptive Activation Functioncited by 4
  26. 2024The effect of Leaky ReLUs on the training and generalization of overparameterized networkscited by 2

Built on by

Not cited yet.