mlsanityData to 7 Oct 2026

READ: Recurrent Adaptation of Large Transformers

arXiv 2305.15348 ↗First posted 24 May 2023cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large-scale Transformers has led to the explosion of many AI applications across Natural Language Processing and Computer Vision tasks. However, fine-tuning all pre-trained model parameters becomes impractical as the model size and number of tasks increase. Parameter-efficient transfer learning (PETL) methods aim to address these challenges. While effective in reducing the number of trainable parameters, PETL methods still require significant energy and computational resources to fine-tune. In this paper, we introduce REcurrent ADaption (READ) -- a lightweight and memory-efficient fine-tuning method -- to overcome the limitations of the current PETL approaches. Specifically, READ inserts a small RNN network alongside the backbone model so that the model does not have to back-propagate through the large backbone network. Through comprehensive empirical evaluation of the GLUE benchmark, we demonstrate READ can achieve a 56% reduction in the training memory consumption and an 84% reduction in the GPU energy usage while retraining high model quality compared to full-tuning. Additionally, the model size of READ does not grow with the backbone model size, making it a highly scalable solution for fine-tuning large Transformers.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

33 of 38 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
21 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018Neural Network Acceptability Judgmentscited by 952
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  13. 2017Large Batch Training of Convolutional Networkscited by 529
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2023Larger language models do in-context learning differentlycited by 386
  16. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  17. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  18. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  19. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  20. 2020Understanding the Difficulty of Training Transformerscited by 210
  21. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 1; 2026: 1.

9 papers.

  1. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  2. 2024DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Modelcited by 8
  3. 2024Sustainable self-supervised learning for speech representationscited by 4
  4. 2024Towards Optimal Adapter Placement for Efficient Transfer Learningcited by 3
  5. 2024Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Modelscited by 2
  6. 2024DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Modelscited by 2
  7. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2
  8. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  9. 2026Selective State-Space Adaptation and Retrieval for Language Model Reasoningcited by 1