mlsanityData to 7 Oct 2026

RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging

arXiv 2510.20479 ↗First posted 23 Oct 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model merging framework for continual learning without access to historical data. RECALL computes inter-model similarity from layer-wise hidden representations over clustered typical samples, and performs adaptive, hierarchical parameter fusion to align knowledge across models. This design enables the preservation of domain-general features in shallow layers while allowing task-specific adaptation in deeper layers. Unlike prior methods that require task labels or incur performance trade-offs, RECALL achieves seamless multi-domain integration and strong resistance to catastrophic forgetting. Extensive experiments across five NLP tasks and multiple continual learning scenarios show that RECALL outperforms baselines in both knowledge retention and generalization, providing a scalable and data-free solution for evolving LLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

36 of 47 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
24 more
  1. 2016Learning without Forgettingcited by 2,465
  2. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2016Progressive Neural Networkscited by 1,332
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  13. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  14. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  15. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  16. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  17. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  18. 2018Selective Experience Replay for Lifelong Learningcited by 239
  19. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  20. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  21. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  22. 2018Adversarial Decomposition of Text Representationcited by 18
  23. 2024Probing Language Models on Their Knowledge Sourcecited by 11
  24. 2023Probing LLMs for Joint Encoding of Linguistic Categoriescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
  2. 2026Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansioncited by 1
  3. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  4. 2025Real Time Detection and Quantitative Analysis of Spurious Forgetting in Continual Learningcited by 0
  5. 2026Hierarchical Wasserstein Merging for Multi-Domain Multi-Task Learning: From Specialists to a Generalistcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  2. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
  4. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuning
  5. 2026Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation
  6. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
  7. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
  8. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation