mlsanityData to 14 Oct 2026

Attention Mechanism with Energy-Friendly Operations

arXiv 2204.13353 ↗First posted 28 Apr 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Attention mechanism has become the dominant module in natural language processing models. It is computationally intensive and depends on massive power-hungry multiplications. In this paper, we rethink variants of attention mechanism from the energy consumption aspects. After reaching the conclusion that the energy costs of several energy-friendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions. Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure. Code is available at: https://github.com/NLP2CT/E-Att.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

21 of 28 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  10. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  11. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  12. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
9 more
  1. 2020Binary Neural Networks: A Surveycited by 190
  2. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  3. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  4. 2019AdderNet: Do We Really Need Multiplications in Deep Learning?cited by 82
  5. 2018Modeling Localness for Self-Attention Networkscited by 43
  6. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 43
  7. 2020AdderSR: Towards Energy Efficient Image Super-Resolutioncited by 32
  8. 2019Context-Aware Self-Attention Networkscited by 25
  9. 2017Deep Semantic Role Labeling with Self-Attentioncited by 23

Built on by

Not cited yet.