mlsanityData to 14 Oct 2026

MoDE: CLIP Data Experts via Clustering

arXiv 2404.16030 ↗First posted 24 Apr 2024cs.CV cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

The success of contrastive language-image pretraining (CLIP) relies on the supervision from the pairing between images and captions, which tends to be noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn a system of CLIP data experts via clustering. Each data expert is trained on one data cluster, being less sensitive to false negative noises in other clusters. At inference time, we ensemble their outputs by applying weights determined through the correlation between task metadata and cluster conditions. To estimate the correlation precisely, the samples in one cluster should be semantically similar, but the number of data experts should still be reasonable for training and inference. As such, we consider the ontology in human language and propose to use fine-grained cluster centers to represent each data expert at a coarse-grained level. Experimental studies show that four CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and OpenCLIP on zero-shot image classification but with less (<35%) training cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly include new data experts. The code is available at https://github.com/facebookresearch/MetaCLIP/tree/main/mode.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

39 of 54 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2017Prototypical Networks for Few-shot Learningcited by 3,015
27 more
  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  10. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  13. 2022Test-Time Training with Masked Autoencoderscited by 212
  14. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  15. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  16. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  17. 2020Support-set bottlenecks for video-text representation learningcited by 170
  18. 2023Hyperbolic Image-Text Representationscited by 136
  19. 2013Learning by Transductioncited by 121
  20. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  21. 2022Few-Shot Object Detection with Fully Cross-Transformercited by 64
  22. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  23. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  24. 2021Partner-Assisted Learning for Few-Shot Image Classificationcited by 36
  25. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  26. 2022TempCLR: Temporal Alignment Representation with Contrastive Learningcited by 15
  27. 2022Global Contrastive Batch Sampling via Optimization on Sample Permutationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 50
  2. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  3. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
  4. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  5. 2024Contextual Document Embeddingscited by 19
  6. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  7. 2025Decentralized Diffusion Modelscited by 13
  8. 2025Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Modelscited by 12
  9. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  10. 2025Seeing What Matters: Empowering CLIP with Patch Generation-to-Selectioncited by 7
  11. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  12. 2025LogiCoL: Logically-Informed Contrastive Learning for Set-based Dense Retrievalcited by 3
10 more
  1. 2025Creating User-steerable Projections with Interactive Semantic Mappingcited by 3
  2. 2025SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Spacecited by 3
  3. 2025Explaining Domain Shifts in Language: Concept erasing for Interpretable Image Classificationcited by 2
  4. 2025Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentationcited by 2
  5. 2024WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalizationcited by 1
  6. 2025Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoderscited by 0
  7. 2025Element2Vec: Build Chemical Element Representation from Text for Property Predictioncited by 0
  8. 2026Decentralized Autoregressive Generationcited by 0
  9. 2026LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Expertscited by 0
  10. 2026AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularizationcited by 0