mlsanityData to 7 Oct 2026

Mitigating Catastrophic Forgetting in Language Transfer via Model Merging

arXiv 2407.08699 ↗First posted 11 Jul 2024cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

As open-weight large language models (LLMs) achieve ever more impressive performances across a wide range of tasks in English, practitioners aim to adapt these models to different languages. However, such language adaptation is often accompanied by catastrophic forgetting of the base model's capabilities, severely limiting the usefulness of the resulting model. We address this issue by proposing Branch-and-Merge (BaM), a new adaptation method based on iteratively merging multiple models, fine-tuned on a subset of the available training data. BaM is based on the insight that this yields lower magnitude but higher quality weight changes, reducing forgetting of the source domain while maintaining learning on the target domain. We demonstrate in an extensive empirical study on Bulgarian and German that BaM can significantly reduce forgetting while matching or even improving target domain performance compared to both standard continued pretraining and instruction finetuning across different model architectures.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 47 of its 76 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

47 of 76 references.

References · 47

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
35 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  12. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  13. 2024LoRA Learns Less and Forgets Lesscited by 313
  14. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  15. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  16. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  17. 2023Mitigating the Alignment Tax of RLHFcited by 181
  18. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  19. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  20. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  21. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  22. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  23. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  24. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  25. 2022Fusing finetuned models for better pretrainingcited by 110
  26. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  27. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  28. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  29. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  30. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  31. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  32. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  33. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  34. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  35. 2024Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtralcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 23; 2026: 8.

36 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  3. 2024ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Modelscited by 26
  4. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  5. 2025SEA-LION: Southeast Asian Languages in One Networkcited by 17
  6. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  7. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  8. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRAcited by 14
  9. 2025Activation-Guided Consensus Merging for Large Language Modelscited by 12
  10. 2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 11
  11. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  12. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
24 more
  1. 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
  2. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  3. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
  4. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  5. 2025Temporal Information Retrieval via Time-Specifier Model Mergingcited by 5
  6. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  7. 2025AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learningcited by 4
  8. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  9. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  10. 2024Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptationcited by 3
  11. 2025Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfercited by 3
  12. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  13. 2026Layer-wise Swapping for Generalizable Multilingual Safetycited by 2
  14. 2025Improving Continual Pre-training Through Seamless Data Packingcited by 1
  15. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  16. 2025Adapting Multilingual Models to Code-Mixed Tasks via Model Mergingcited by 1
  17. 2025LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soupscited by 1
  18. 2026ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Mergingcited by 1
  19. 2025Joint Flashback Adaptation for Forgetting-Resistant Instruction Tuningcited by 0
  20. 2026Awakening the Sleeping Agent: Lean-Specific Agentic Data Reactivates General Tool Use in Goedel Provercited by 0
  21. 2026Jupiter-N Technical Reportcited by 0
  22. 2026Leveraging Instruction Tuning and Merging for Reasoning Model Adaptationcited by 0
  23. 2026Continual Learning in Transitioncited by 0
  24. 2026The evolution of sex for artificial intelligence: a population-genetic framework for multigenerational model populationscited by 0