mlsanityData to 7 Oct 2026

CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling

arXiv 2409.19291 ↗First posted 28 Sep 2024cs.CV cs.AIcited by 23

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in multimodal intelligence. However, recent studies discovered that CLIP can only encode one aspect of the feature space, leading to substantial information loss and indistinctive features. To mitigate this issue, this paper introduces a novel strategy that fine-tunes a series of complementary CLIP models and transforms them into a CLIP-MoE. Specifically, we propose a model-agnostic Diversified Multiplet Upcycling (DMU) framework for CLIP. Instead of training multiple CLIP models from scratch, DMU leverages a pre-trained CLIP and fine-tunes it into a diverse set with highly cost-effective multistage contrastive learning, thus capturing distinct feature subspaces efficiently. To fully exploit these fine-tuned models while minimizing computational overhead, we transform them into a CLIP-MoE, which dynamically activates a subset of CLIP experts, achieving an effective balance between model capacity and computational cost. Comprehensive experiments demonstrate the superior performance of CLIP-MoE across various zero-shot retrieval, zero-shot image classification tasks, and downstream Multimodal Large Language Model (MLLM) benchmarks when used as a vision encoder.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 49 of its 65 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

49 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
36 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2024Mixtral of Expertscited by 1,785
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  13. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  15. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  16. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  17. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  18. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  19. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  20. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  21. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  22. 2023Demystifying CLIP Datacited by 265
  23. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  24. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  25. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  26. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  27. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  28. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  29. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  30. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  31. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Modelscited by 23
  32. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  33. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  34. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Expertscited by 7
  35. 2022Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrievalcited by 3
  36. 2024Intelligent Text-Conditioned Music Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  3. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  4. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  5. 2025SmartCLIP: Modular Vision-language Alignment with Identification Guaranteescited by 14
  6. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  7. 2025From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibrationcited by 8
  8. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  9. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 6
  10. 2025CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcyclingcited by 5
  11. 2025Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoderscited by 5
  12. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Freecited by 4
11 more
  1. 2026MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Modelscited by 3
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  3. 2025StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editingcited by 1
  4. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  5. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  6. 2025MIN-Merging: Merge the Important Neurons for Model Mergingcited by 0
  7. 2026Micro-AU CLIP: Fine-Grained Contrastive Learning from Local Independence to Global Dependency for Micro-Expression Action Unit Detectioncited by 0
  8. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  9. 2026TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separationcited by 0
  10. 2026FaceMoE: Mixture of Experts for Low-Resolution Face Recognitioncited by 0
  11. 2026TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  2. 2024Diffusion Feedback Helps CLIP See Better
  3. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
  4. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination
  5. 2025Meta CLIP 2: A Worldwide Scaling Recipe
  6. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  7. 2024Toward a Holistic Evaluation of Robustness in CLIP Models
  8. 2025CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation