Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 52 of its 88 references that are in the map. First posted on arXiv 10 Nov 2025.
Builds on
52 of 88 references.
References · 52
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
- 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
- 2015Teaching Machines to Read and Comprehendcited by 1,049
38 more
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2017Exploring Generalization in Deep Learningcited by 600
- 2023The Falcon Series of Open Language Modelscited by 535
- 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
- 2022Autoformalization with Large Language Modelscited by 263
- 2018Multi-Adversarial Domain Adaptationcited by 196
- 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
- 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 90
- 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
- 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
- 2023MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Modelscited by 63
- 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 58
- 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
- 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
- 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
- 2024OmniGuard: Hybrid Manipulation Localization via Augmented Versatile Deep Image Watermarkingcited by 35
- 2024M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Frameworkcited by 32
- 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
- 2024Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domainscited by 30
- 2024Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrievalcited by 30
- 2024Improving Data Efficiency via Curating LLM-Driven Rating Systemscited by 29
- 2025Removal of Hallucination on Hallucination: Debate-Augmented RAGcited by 29
- 2021Understanding the Dynamics of DNNs Using Graph Modularitycited by 26
- 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Modelcited by 25
- 2024Swift Sampler: Efficient Learning of Sampler by 10 Parameterscited by 25
- 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
- 2023RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Modelscited by 24
- 2023SyreaNet: A Physically Guided Underwater Image Enhancement Framework Integrating Synthetic and Real Imagescited by 23
- 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
- 2024A Generic Layer Pruning Method for Signal Modulation Recognition Deep Learning Modelscited by 18
- 2024YOLO-PPA based Efficient Traffic Sign Detection for Cruise Control in Autonomous Drivingcited by 16
- 2025Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansioncited by 15
- 2025AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Modelscited by 14
- 20254D Driving Scene Generation With Stereo Forcingcited by 8
- 2025C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detectioncited by 5
- 2025Renaissance of RNNs in Streaming Clinical Time Series: Compact Recurrence Remains Competitive with Transformerscited by 5
- 2025Curiosity Meets Cooperation: A Game-Theoretic Approach to Long-Tail Multi-Label Learningcited by 4
- 2025DOA: A Degeneracy Optimization Agent with Adaptive Pose Compensation Capability based on Deep Reinforcement Learningcited by 3
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.