Merging Language and Domain Specific Models: The Impact on Technical Vocabulary Acquisition
Not on the map. Why?
Abstract (arXiv)
Advancements in Natural Language Processing have enabled specialized language models, but integrating domain-specific knowledge into general-purpose models in multilingual settings remains challenging, particularly for technical vocabulary. This paper investigates the integration of technical vocabulary in merged language models and explores the knowledge transfer mechanisms involved when combining a general-purpose language-specific model with a domain-specific model, focusing on the resulting model's comprehension of technical jargon. Our experiments analyze the impact of this merging process on the target model's proficiency in handling specialized terminology. We present a quantitative evaluation of the performance of the merged model, comparing it with that of the individual constituent models. The findings offer insights into the effectiveness of different model merging methods for enhancing domain-specific knowledge and highlight potential challenges and future directions in leveraging these methods for cross-lingual knowledge transfer in Natural Language Processing.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 17 Feb 2025.
Builds on
25 of 36 references.
References · 25
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
- 2019How multilingual is Multilingual BERT?cited by 645
- 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
- 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
- 2024Evolutionary Optimization of Model Merging Recipescited by 208
13 more
- 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
- 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
- 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
- 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
- 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
- 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 42
- 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Modelscited by 18
- 2022Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structurecited by 17
- 2024Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluationcited by 16
- 2024JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Modelscited by 8
- 2024Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancementcited by 4
- 2024Tagengo: A Multilingual Chat Datasetcited by 3
- 2023Cross-lingual Knowledge Transfer via Distillation for Multilingual Information Retrievalcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.