mlsanityData to 7 Oct 2026

EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

arXiv 2202.07959 ↗First posted 16 Feb 2022cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers. Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM -- the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

34 of 48 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
22 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2016Sequence-Level Knowledge Distillationcited by 761
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  4. 2017Convolutional Sequence to Sequence Learningcited by 550
  5. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  6. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  7. 2018Scaling Neural Machine Translationcited by 204
  8. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  9. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  10. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  11. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  12. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  13. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  14. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
  15. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  16. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  17. 2019Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generationcited by 57
  18. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  19. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  20. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  21. 2020Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correctioncited by 24
  22. 2021Improving Sequence-to-Sequence Pre-training via Sequence Span Rewritingcited by 13

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 4; 2024: 6; 2025: 6.

21 papers.

Sort:

By area:

  1. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  2. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  3. 2025GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphscited by 17
  4. 2024PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRAcited by 16
  5. 2023One Wide Feedforward is All You Needcited by 15
  6. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  7. 2024The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Promptscited by 13
  8. 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11
  9. 2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
  10. 2022MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformerscited by 5
  11. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  12. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
9 more
  1. 2025Compact Neural TTS Voices for Accessibilitycited by 2
  2. 2023MobileNMT: Enabling Translation in 15MB and 30mscited by 1
  3. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  4. 2025Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutionscited by 1
  5. 2025Context-Selective State Space Models: Feedback is All You Needcited by 1
  6. 2022Real-time Speech Interruption Analysis: From Cloud to Client Deploymentcited by 0
  7. 2023Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddingscited by 0
  8. 2024SPA: Towards A Computational Friendly Cloud-Base and On-Devices Collaboration Seq2seq Personalized Generation with Casual Inferencecited by 0
  9. 2025EdgeAttNet: Towards Barb-Aware Filament Segmentationcited by 0