mlsanityData to 7 Oct 2026

SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models

arXiv 2401.08295 ↗First posted 16 Jan 2024cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

The continual learning (CL) ability is vital for deploying large language models (LLMs) in the dynamic world. Existing methods devise the learning module to acquire task-specific knowledge with parameter-efficient tuning (PET) block and the selection module to pick out the corresponding one for the testing input, aiming at handling the challenges of catastrophic forgetting and knowledge transfer in CL. However, these methods tend to address only one of the challenges, ignoring the potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfer simultaneously. To this end, we propose a novel Shared Attention Framework (SAPT), to align the PET learning and selection via the Shared Attentive Learning & Selection module. Extensive Experiments on two CL benchmarks demonstrate the superiority of SAPT. Moreover, SAPT consistently demonstrates its superiority when we scale it to different model sizes (from 770M to 13B), different model architectures (T5 and LLaMA-2) and unseen tasks.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

56 of 79 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2016Layer Normalizationcited by 5,579
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2016Learning without Forgettingcited by 2,465
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2017Continual Learning with Deep Generative Replaycited by 1,152
  11. 2018Experience Replay for Continual Learningcited by 1,074
  12. 2021Learning to Prompt for Continual Learningcited by 938
  13. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  14. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  15. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  16. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  17. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  18. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  19. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  20. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  21. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  22. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  23. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  24. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  25. 2018Selective Experience Replay for Lifelong Learningcited by 239
  26. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  27. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  28. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  29. 2019Episodic Memory in Lifelong Language Learningcited by 188
  30. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  31. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  32. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  33. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  34. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  35. 2020GLUCOSE: GeneraLized and COntextualized Story Explanationscited by 85
  36. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  37. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  38. 2018A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literaturecited by 79

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 13.

37 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2024Continual Learning for Large Language Models: A Surveycited by 144
  3. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  4. 2024Unleashing the Power of Continual Learning on Non-Centralized Devices: A Surveycited by 23
  5. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  6. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  7. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  8. 2025Continual Multimodal Contrastive Learningcited by 19
  9. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  10. 2025Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulationcited by 13
  11. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 12
  12. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
25 more
  1. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  2. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  3. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  4. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  5. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  6. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  7. 2026ELLA: Efficient Lifelong Learning for Adapters in Large Language Modelscited by 3
  8. 2024Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspectivecited by 2
  9. 2025K-DeCore: Facilitating Knowledge Transfer in Continual Structured Knowledge Reasoning via Knowledge Decouplingcited by 2
  10. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  11. 2025Parameter Importance-Driven Continual Learning for Foundation Modelscited by 2
  12. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learningcited by 2
  13. 2025Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learningcited by 1
  14. 2026Turning Back Without Forgetting: Selective Backward Refinement for Parameter-Efficient Continual Learningcited by 1
  15. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  16. 2025Mixtures of SubExperts for Large Language Continual Learningcited by 0
  17. 2025PPSEBM: An Energy-Based Model with Progressive Parameter Selection for Continual Learningcited by 0
  18. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0
  19. 2026Attribution-Guided Continual Learning for Large Language Modelscited by 0
  20. 2026CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learningcited by 0
  21. 2026CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMscited by 0
  22. 2026Continual Learning in Transitioncited by 0
  23. 2026Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learningcited by 0
  24. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 0
  25. 2026ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMscited by 0