# EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1909.01939 · HTML: https://mlsanity.com/paper/1909.01939/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality](https://mlsanity.com/paper/2207.07036/) (2022)
- [Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2305.09212/) (2023)
- [Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10563/) (2023)
- [Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries](https://mlsanity.com/paper/2102.08065/) (2021)
- [Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition](https://mlsanity.com/paper/2202.11374/) (2022)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [Hyperspectral Image Denoising via Spatial-Spectral Recurrent Transformer](https://mlsanity.com/paper/2401.03885/) (2023)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10567/) (2023)
- [Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2111.03993/) (2021)
- [Complementary Matrix-Gated QKAN Fast-Weight Programmers for Quantum Dynamics Forecasting](https://mlsanity.com/paper/2607.27945/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are capable of modeling temporal dependencies of complex sequential data. In general, current available structures of RNNs tend to concentrate on controlling the contributions of current and previous information. However, the exploration of different importance levels of different elements within an input vector is always ignored. We propose a simple yet effective Element-wise-Attention Gate (EleAttG), which can be easily added to an RNN block (e.g. all RNN neurons in an RNN layer), to empower the RNN neurons to have attentiveness capability. For an RNN block, an EleAttG is used for adaptively modulating the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Instead of modulating the input as a whole, the EleAttG modulates the input at fine granularity, i.e., element-wise, and the modulation is content adaptive. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to different tasks including the action recognition, from both skeleton-based data and RGB videos, gesture recognition, and sequential MNIST classification. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly improves the power of RNNs.