mlsanityData to 7 Oct 2026

Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model

arXiv 2406.19905 ↗First posted 28 Jun 2024cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The Mixture-of-Experts (MoE) has gained increasing attention in studying Large Vision-Language Models (LVLMs). It uses a sparse model to replace the dense model, achieving comparable performance while activating fewer parameters during inference, thus significantly reducing the inference cost. Existing MoE methods in LVLM encourage different experts to specialize in different tokens, and they usually employ a router to predict the routing of each token. However, the router is not optimized concerning distinct parameter optimization directions generated from tokens within an expert. This may lead to severe interference between tokens within an expert. To address this problem, we propose to use the token-level gradient analysis to Solving Token Gradient Conflict (STGC) in this paper. Specifically, we first use token-level gradients to identify conflicting tokens in experts. After that, we add a regularization loss tailored to encourage conflicting tokens routing from their current experts to other experts, for reducing interference between tokens within an expert. Our method can serve as a plug-in for diverse LVLM methods, and extensive experimental results demonstrate its effectiveness. The code will be publicly available at https://github.com/longrongyang/STGC.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 81 of its 104 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

81 of 104 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May
  2. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  3. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec
  7. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
1 more
  1. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 68

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021Convex optimizationcited by 2,479
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2024Mixtral of Expertscited by 1,785
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  15. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  16. 2018Neural Network Acceptability Judgmentscited by 952
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  18. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  19. 2016A Diagram Is Worth A Dozen Imagescited by 793
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  21. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  22. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  23. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  24. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  25. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  26. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  27. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  28. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  29. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  30. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  31. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  32. 2023From Sparse to Soft Mixtures of Expertscited by 245
  33. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  34. 2024Mixture of LoRA Expertscited by 179
  35. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  36. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  37. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  38. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 4.

11 papers.

  1. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learningcited by 20
  2. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  3. 2025Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detectioncited by 8
  4. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  5. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  6. 2025GradientSpace: Unsupervised Data Clustering for Improved Instruction Tuningcited by 2
  7. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  8. 2026DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimizationcited by 2
  9. 2025Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularitycited by 0
  10. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 0
  11. 2026OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
  2. 2025Advancing Expert Specialization for Better MoE
  3. 2025Sigma-MoE-Tiny Technical Report
  4. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  6. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  7. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  8. 2025Mixture of Experts in Large Language Models