# λ-Scaled-Attention: A Novel Fast Attention Mechanism for Efficient Modeling of Protein Sequences

arXiv: https://arxiv.org/abs/2201.02912 · HTML: https://mlsanity.com/paper/2201.02912/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 43 references that are in the map.
- First posted on arXiv 9 Jan 2022.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [ProLanGO: Protein Function Prediction Using Neural~Machine Translation Based on a Recurrent Neural Network](https://mlsanity.com/paper/1710.07016/) (2017)
- [Deep Robust Framework for Protein Function Prediction using Variable-Length Protein Sequences](https://mlsanity.com/paper/1811.01338/) (2018)

## Abstract (arXiv)

Attention-based deep networks have been successfully applied on textual data in the field of NLP. However, their application on protein sequences poses additional challenges due to the weak semantics of the protein words, unlike the plain text words. These unexplored challenges faced by the standard attention technique include (i) vanishing attention score problem and (ii) high variations in the attention distribution. In this regard, we introduce a novel λ-scaled attention technique for fast and efficient modeling of the protein sequences that addresses both the above problems. This is used to develop the λ-scaled attention network and is evaluated for the task of protein function prediction implemented at the protein sub-sequence level. Experiments on the datasets for biological process (BP) and molecular function (MF) showed significant improvements in the F1 score values for the proposed λ-scaled attention technique over its counterpart approach based on the standard attention technique (+2.01% for BP and +4.67% for MF) and state-of-the-art ProtVecGen-Plus approach (+2.61% for BP and +4.20% for MF). Further, fast convergence (converging in half the number of epochs) and efficient learning (in terms of very low difference between the training and validation losses) were also observed during the training process.