mlsanityData to 14 Oct 2026

LM-Cocktail: Resilient Tuning of Language Models via Model Merging

arXiv 2311.13534 ↗First posted 22 Nov 2023cs.CL cs.AI cs.IRcited by 44

Not on the map. Why?

Abstract (arXiv)

The pre-trained language models are continually fine-tuned to better support downstream applications. However, this operation may result in significant performance degeneration on general tasks beyond the targeted domain. To overcome this problem, we propose LM-Cocktail which enables the fine-tuned model to stay resilient in general perspectives. Our method is conducted in the form of model merging, where the fine-tuned language model is merged with the pre-trained base model or the peer models from other domains through weighted average. Despite simplicity, LM-Cocktail is surprisingly effective: the resulted model is able to achieve a strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. We conduct comprehensive experiments with LLama and BGE model on popular benchmarks, including FLAN, MMLU, MTEB, whose results validate the efficacy of our proposed method. The code and checkpoints are available at https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

40 of 56 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  12. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
28 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  10. 2017Continual Learning with Deep Generative Replaycited by 1,158
  11. 2018Experience Replay for Continual Learningcited by 1,095
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
  13. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  14. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  15. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  16. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  17. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  18. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  19. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  20. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  21. 2019Model Fusion via Optimal Transportcited by 219
  22. 2022Patching open-vocabulary models by interpolating weightscited by 190
  23. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  24. 2017Encoder Based Lifelong Learningcited by 102
  25. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 75
  26. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  27. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  28. 2020Versatile Black-Box Optimizationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 18; 2026: 8.

44 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  3. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
  4. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  5. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  6. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  7. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  8. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  9. 2024KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checkingcited by 26
  10. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  11. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledgecited by 20
  12. 2025Fun-Audio-Chat Technical Reportcited by 19
32 more
  1. 2024HFT: Half Fine-Tuning for Large Language Modelscited by 18
  2. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  3. 2024Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuningcited by 10
  4. 2025Granite Embedding Modelscited by 8
  5. 2026HumanLLM: Towards Personalized Understanding and Simulation of Human Naturecited by 8
  6. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusioncited by 8
  7. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  8. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  9. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  10. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  11. 2025HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Mergingcited by 6
  12. 2025Temporal Information Retrieval via Time-Specifier Model Mergingcited by 5
  13. 2025RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Mergingcited by 5
  14. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  15. 2024Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Mergingcited by 4
  16. 2025Efficient Compositional Multi-tasking for On-device Large Language Modelscited by 4
  17. 2025K-Merge: Online Continual Merging of Adapters for On-device Large Language Modelscited by 4
  18. 2024Activated Parameter Locating via Causal Intervention for Model Mergingcited by 3
  19. 2024Parameter-Efficient Interventions for Enhanced Model Mergingcited by 3
  20. 2025Unified Multi-Task Learning & Model Fusion for Efficient Language Model Guardrailingcited by 3
  21. 2025Purifying Task Vectors in Knowledge-Aware Subspace for Model Mergingcited by 3
  22. 2024MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Mergingcited by 2
  23. 2025Robust Learning of Diverse Code Editscited by 2
  24. 2025MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge through Group Direct Preference Optimizationcited by 2
  25. 2026Multi-Stage Evolutionary Model Merging with Meta Data Driven Curriculum Learning for Sentiment-Specialized Large Language Modelingcited by 2
  26. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  27. 2026Intent-aligned Formal Specification Synthesis via Traceable Refinementcited by 2
  28. 2024REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Modelscited by 1
  29. 2025Behave Your Motion: Habit-preserved Cross-category Animal Motion Transfercited by 1
  30. 2026Modular Expert Merging for Biomedical Retrievalcited by 0
  31. 2026Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflowscited by 0
  32. 2026Lychee-Reranker: Fast but Not Late Interaction for Compressed Document Rerankingcited by 0