mlsanityData to 14 Oct 2026

Supplementary Material: Implementation and Experiments for GAU-based Model

arXiv 2205.05842 ↗First posted 12 May 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In February this year Google proposed a new Transformer variant called FLASH, which has a faster speed, lower VRAM footprint and better performance. This is achieved by designing a performant layer named GAU (Gated Attention Unit), which combines the Attention layer and FFN. In this paper, some implementation details are re-analyzed both theoretically and practically. We then propose a novel GAU-based model and pre-train it on a Chinese corpus. Results of the CLUE benchmark show that our model achieves a dev average score of 75.02, 1% higher than RoFormerV1 and being 45% faster, which is also competitive with RoFormerV2.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

18 of 24 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2020Rethinking Attention with Performerscited by 1,601
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  12. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
6 more
  1. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  2. 2022Transformer Quality in Linear Timecited by 191
  3. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  4. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  5. 2021Finetuning Pretrained Transformers into RNNscited by 96
  6. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87

Built on by

Not cited yet.