Neuromodulation Gated Transformer
Not on the map. Why?
Abstract (arXiv)
We introduce a novel architecture, the Neuromodulation Gated Transformer (NGT), which is a simple implementation of neuromodulation in transformers via a multiplicative effect. We compare it to baselines and show that it results in the best average performance on the SuperGLUE benchmark validation sets.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 5 May 2023.
Builds on
16 of 36 references.
References · 16
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
- 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
- 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
4 more
- 2020Learning to Continually Learncited by 82
- 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
- 2018Introducing Neuromodulation in Deep Neural Networks to Learn Adaptive Behaviourscited by 16
- 2017Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networkscited by 9
Built on by
Not cited yet.