mlsanityData to 7 Oct 2026

Gaussian Error Linear Units (GELUs)

arXiv 1606.08415 ↗First posted 27 Jun 2016cs.LGcited by 3,431

Not on the map. Why?

Cited by 163 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is xPhi(x), where Phi(x) the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs (x1x>0). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

Key facts. Cited by 3,431 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 304; 2022: 478; 2023: 692; 2024: 745; 2025: 665; 2026: 545.

3,431 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  11. 2021Vision Transformers for Dense Predictioncited by 1,789
  12. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
38 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  12. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  14. 2022A Generalist Agentcited by 802
  15. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  16. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  17. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  18. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  19. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  20. 2022Galactica: A Large Language Model for Sciencecited by 736
  21. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  22. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  23. 2022Transformers learn in-context by gradient descentcited by 714
  24. 2020Aligning AI With Shared Human Valuescited by 706
  25. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  26. 2022Simple Baselines for Image Restorationcited by 701
  27. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  28. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  29. 2023A Comprehensive Overview of Large Language Modelscited by 639
  30. 2021Transformer in Transformercited by 632
  31. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  32. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  33. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  34. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  35. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  36. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  37. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  38. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536

50 most cited of 3,431 All 3,431 (JSON)