mlsanityData to 7 Oct 2026

Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts

arXiv 2406.11256 ↗First posted 17 Jun 2024cs.CLcited by 19

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) models have shown remarkable capability in instruction tuning, especially when the number of tasks scales. However, previous methods simply merge all training tasks (e.g. creative writing, coding, and mathematics) and apply fixed sampling weights, without considering the importance of different tasks as the model training state changes. In this way, the most helpful data cannot be effectively distinguished, leading to suboptimal model performance. To reduce the potential redundancies of datasets, we make the first attempt and propose a novel dynamic data mixture for MoE instruction tuning. Specifically, inspired by MoE's token routing preference, we build dataset-level representations and then capture the subtle differences among datasets. Finally, we propose to dynamically adjust the sampling weight of datasets by their inter-redundancies, thus maximizing global performance under a limited training budget. The experimental results on two MoE models demonstrate the effectiveness of our approach on both downstream knowledge & reasoning tasks and open-ended queries. Code and models are available at https://github.com/Spico197/MoE-SFT .

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 31 of its 54 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

31 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2023Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning22 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
14 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2023LIMA: Less Is More for Alignmentcited by 1,080
  3. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  4. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  9. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  10. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  11. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  12. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  13. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  14. 2023LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigmscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 4.

19 papers.

Sort:

By area:

  1. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  2. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  3. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  4. 2024Learning to Refuse: Towards Mitigating Privacy Risks in LLMscited by 22
  5. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  6. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  7. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  8. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  9. 2025DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collectionscited by 6
  10. 2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Modelscited by 4
  11. 2025Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuningcited by 3
  12. 2025ADAPT: Learning Task Mixtures for Budget-Constrained Instruction Tuningcited by 3
7 more
  1. 2025DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agentscited by 2
  2. 2026mSFT: Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFTcited by 2
  3. 2025HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Modelscited by 1
  4. 2025StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editingcited by 1
  5. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  6. 2025MIN-Merging: Merge the Important Neurons for Model Mergingcited by 0
  7. 2026Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  3. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
  4. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  5. 2025Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
  6. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  7. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  8. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models