mlsanityData to 7 Oct 2026

GLU Variants Improve Transformer

arXiv 2002.05202 ↗First posted 12 Feb 2020cs.LG cs.NE stat.MLcited by 1,724

Not on the map. Why?

Cited by 130 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

Key facts. Cited by 1,724 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 49; 2023: 133; 2024: 415; 2025: 496; 2026: 612.

1,724 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
38 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  3. 2024VMamba: Visual State Space Modelcited by 955
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2025Large Language Diffusion Modelscited by 872
  8. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  9. 2022A Generalist Agentcited by 802
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  12. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  13. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  14. 2024Yi: Open Foundation Models by 01.AIcited by 706
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2022Simple Baselines for Image Restorationcited by 701
  17. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  18. 2023A Comprehensive Overview of Large Language Modelscited by 639
  19. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  20. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  21. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  22. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  23. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  24. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  25. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  26. 2023The Falcon Series of Open Language Modelscited by 526
  27. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  28. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  29. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  30. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  31. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  32. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  33. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  34. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  35. 2021A Survey of Transformerscited by 399
  36. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  37. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  38. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377

50 most cited of 1,724 All 1,724 (JSON)