# On the Role of Activation Functions in EEG-To-Text Decoder

arXiv: https://arxiv.org/abs/2410.12572 · HTML: https://mlsanity.com/paper/2410.12572/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 55 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation](https://mlsanity.com/paper/2309.14030/) (2023)
- [Learning Activation Functions: A new paradigm for understanding Neural Networks](https://mlsanity.com/paper/1906.09529/) (2019)
- [Efficacy of Transformer Networks for Classification of Raw EEG Data](https://mlsanity.com/paper/2202.05170/) (2022)
- [Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals](https://mlsanity.com/paper/1607.03502/) (2016)
- [Deep Convolutional Neural Network Applied to Electroencephalography: Raw Data vs Spectral Features](https://mlsanity.com/paper/2105.04762/) (2021)
- [Reading Task Classification Using EEG and Eye-Tracking Data](https://mlsanity.com/paper/2112.06310/) (2021)
- [Prediction of the Realisation of an Information Need: An EEG Study](https://mlsanity.com/paper/2406.08105/) (2024)

## Abstract (arXiv)

In recent years, much interdisciplinary research has been conducted exploring potential use cases of neuroscience to advance the field of information retrieval. Initial research concentrated on the use of fMRI data, but fMRI was deemed to be not suitable for real-world applications, and soon, research shifted towards using EEG data. In this paper, we try to improve the original performance of a first attempt at generating text using EEG by focusing on the less explored area of optimising neural network performance. We test a set of different activation functions and compare their performance. Our results show that introducing a higher degree polynomial activation function can enhance model performance without changing the model architecture. We also show that the learnable 3rd-degree activation function performs better on the 1-gram evaluation compared to a 3rd-degree non-learnable function. However, when evaluating the model on 2-grams and above, the polynomial function lacks in performance, whilst the leaky ReLU activation function outperforms the baseline.