Delving Deeper Into Astromorphic Transformers
Not on the map. Why?
Abstract (arXiv)
Preliminary attempts at incorporating the critical role of astrocytes - cells that constitute more than 50% of human brain cells - in brain-inspired neuromorphic computing remain in infancy. This paper seeks to delve deeper into various key aspects of neuron-synapse-astrocyte interactions to mimic self-attention mechanisms in Transformers. The cross-layer perspective explored in this work involves bioplausible modeling of Hebbian and presynaptic plasticities in neuron-astrocyte networks, incorporating effects of non-linearities and feedback along with algorithmic formulations to map the neuron-astrocyte computations to self-attention mechanism and evaluating the impact of incorporating bio-realistic effects from the machine learning application side. Our analysis on sentiment and image classification tasks (IMDB and CIFAR10 datasets) highlights the advantages of Astromorphic Transformers, offering improved accuracy and learning speed. Furthermore, the model demonstrates strong natural language generation capabilities on the WikiText-2 dataset, achieving better perplexity compared to conventional models, thus showcasing enhanced generalization and stability across diverse machine learning tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 83 references that are in the map. First posted on arXiv 18 Dec 2023.
Builds on
29 of 83 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
- 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
- 2019Stand-Alone Self-Attention in Vision Modelscited by 542
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
- 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
- 2018Deep Learning in Spiking Neural Networkscited by 263
- 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 245
17 more
- 2017SuperSpike: Supervised learning in multi-layer spiking neural networkscited by 182
- 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
- 2019An Attentive Survey of Attention Modelscited by 124
- 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
- 2016Supervised learning based on temporal coding in spiking neural networkscited by 76
- 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
- 2020Learning Associative Inference Using Fast Weight Memorycited by 51
- 2019Language Models with Transformerscited by 36
- 2018Direct Output Connection for a High-Rank Language Modelcited by 11
- 2021IMPULSE: A 65nm Digital Compute-in-Memory Macro with Fused Weights and Membrane Potential for Spike-based Sequential Learning Taskscited by 6
- 2020On the Self-Repair Role of Astrocytes in STDP Enabled Unsupervised SNNscited by 5
- 2020An Astrocyte-Modulated Neuromorphic Central Pattern Generator for Hexapod Robot Locomotion on Intel's Loihicited by 4
- 2021BioLeaF: A Bio-plausible Learning Framework for Training of Spiking Neural Networkscited by 3
- 2019Deep Residual Output Layers for Neural Language Generationcited by 2
- 2020Emulation of Astrocyte Induced Neural Phase Synchrony in Spin-Orbit Torque Oscillator Neuronscited by 1
- 2022Text Classification in Memristor-based Spiking Neural Networkscited by 1
- 2022Astromorphic Self-Repair of Neuromorphic Hardware Systemscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.