mlsanityData to 14 Oct 2026

Merge to Mix: Mixing Datasets via Model Merging

arXiv 2505.16066 ↗First posted 21 May 2025cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Mixing datasets for fine-tuning large models (LMs) has become critical for maximizing performance on downstream tasks. However, composing effective dataset mixtures typically relies on heuristics and trial-and-error, often requiring multiple fine-tuning runs to achieve the desired outcome. We propose a novel method, textitMerge to Mix, that accelerates composing dataset mixtures through model merging. Model merging is a recent technique that combines the abilities of multiple individually fine-tuned LMs into a single LM by using a few simple arithmetic operations. Our key insight is that merging models individually fine-tuned on each dataset in a mixture can effectively serve as a surrogate for a model fine-tuned on the entire mixture. Merge to Mix leverages this insight to accelerate selecting dataset mixtures without requiring full fine-tuning on each candidate mixture. Our experiments demonstrate that Merge to Mix surpasses state-of-the-art methods in dataset selection for fine-tuning LMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 56 of its 67 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

56 of 67 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
38 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  4. 2023Textbooks Are All You Needcited by 569
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  7. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  8. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  11. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  12. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  13. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  14. 2024A Survey on Data Selection for Language Modelscited by 236
  15. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  16. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  17. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  18. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  19. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  20. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  21. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  22. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  23. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  24. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  25. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  26. 2024Towards Safer Large Language Models through Machine Unlearningcited by 139
  27. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  28. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  29. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
  30. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  31. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  32. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  33. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 58
  34. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  35. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  36. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  37. 2024mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Modelscited by 33
  38. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  2. 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
  3. 2025Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviorscited by 3
  4. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  5. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimizationcited by 1
  6. 2026Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timecited by 0
  7. 2026Efficient Decentralized Multi-task Dataset Valuation via Model Mergingcited by 0