# DATA: Decomposed Attention-based Task Adaptation for Rehearsal-Free Continual Learning

arXiv: https://arxiv.org/abs/2502.11482 · HTML: https://mlsanity.com/paper/2502.11482/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2025.

## Built on by (most cited first, 7 of 7)

- [AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning](https://mlsanity.com/paper/2509.17348/) (2025)
- [SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning](https://mlsanity.com/paper/2511.22367/) (2025)
- [ELLA: Efficient Lifelong Learning for Adapters in Large Language Models](https://mlsanity.com/paper/2601.02232/) (2026)
- [Parameter Importance-Driven Continual Learning for Foundation Models](https://mlsanity.com/paper/2511.15375/) (2025)
- [CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models](https://mlsanity.com/paper/2510.13008/) (2025)
- [SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models](https://mlsanity.com/paper/2605.25525/) (2026)
- [Continual Learning for Sequential Personalization of Small Language Models: A Stability Monitoring Analysis](https://mlsanity.com/paper/2606.27634/) (2026)

## Abstract (arXiv)

Continual learning (CL) is essential for Large Language Models (LLMs) to adapt to evolving real-world demands, yet they are susceptible to catastrophic forgetting (CF). While traditional CF solutions rely on expensive data rehearsal, recent rehearsal-free methods employ model-based and regularization-based strategies to address this issue. However, these approaches often neglect the model's plasticity, which is crucial to achieving optimal performance on newly learned tasks. Consequently, a key challenge in CL is striking a balance between preserving plasticity and mitigating CF. To tackle this challenge, we propose the Decomposed Attention-based Task Adaptation (DATA), which explicitly decouples and learns both task-specific and task-shared knowledge using high-rank and low-rank task adapters (e.g., LoRAs). For new tasks, DATA dynamically adjusts the weights of adapters of different ranks based on their relevance and distinction from previous tasks, allowing the model to acquire new task-specific skills while effectively retaining previously learned knowledge. Specifically, we implement a decomposed component weighting strategy comprising learnable components that collectively generate attention-based weights, allowing the model to integrate and utilize diverse knowledge from each DATA. Extensive experiments on three widely used benchmarks demonstrate that our proposed method achieves state-of-the-art performance. Notably, our approach significantly enhances model plasticity and mitigates CF by extending learnable components and employing stochastic restoration during training iterations.