Improving Autoregressive NLP Tasks via Modular Linearized Attention
Not on the map. Why?
Abstract (arXiv)
Various natural language processing (NLP) tasks necessitate models that are efficient and small based on their ultimate application at the edge or in other resource-constrained environments. While prior research has reduced the size of these models, increasing computational efficiency without considerable performance impacts remains difficult, especially for autoregressive tasks. This paper proposes modular linearized attention (MLA), which combines multiple efficient attention mechanisms, including cosFormer, to maximize inference quality while achieving notable speedups. We validate this approach on several autoregressive NLP tasks, including speech-to-text neural machine translation (S2T NMT), speech-to-text simultaneous translation (SimulST), and autoregressive text-to-spectrogram, noting efficiency gains on TTS and competitive performance for NMT and SimulST during training and inference.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 17 Apr 2023.
Builds on
37 of 47 references.
References · 37
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2020Big Bird: Transformers for Longer Sequencescited by 1,684
- 2020Reformer: The Efficient Transformercited by 1,668
- 2020Rethinking Attention with Performerscited by 1,601
- 2019Generating Long Sequences with Sparse Transformerscited by 1,546
25 more
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2018Image Transformercited by 772
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
- 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
- 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
- 2021Random Feature Attentioncited by 354
- 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
- 2019Revealing the Dark Secrets of BERTcited by 301
- 2018Adaptive Input Representations for Neural Language Modelingcited by 226
- 2022cosFormer: Rethinking Softmax in Attentioncited by 213
- 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
- 2019Adaptive Attention Span in Transformerscited by 149
- 2020ProGen: Language Modeling for Protein Generationcited by 128
- 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
- 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
- 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
- 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
- 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
- 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
- 2022Neural Architecture Search on Efficient Transformers and Beyondcited by 17
- 2021RankNAS: Efficient Neural Architecture Search by Pairwise Rankingcited by 12
- 2022Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentationcited by 12
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.