mlsanityData to 7 Oct 2026

Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts

arXiv 2402.05382 ↗First posted 8 Feb 2024cs.CV cs.LGcited by 16

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Masked Autoencoder~(MAE) is a prevailing self-supervised learning method that achieves promising results in model pre-training. However, when the various downstream tasks have data distributions different from the pre-training data, the semantically irrelevant pre-training information might result in negative transfer, impeding MAE's scalability. To address this issue, we propose a novel MAE-based pre-training paradigm, Mixture of Cluster-conditional Experts (MoCE), which can be trained once but provides customized pre-training models for diverse downstream tasks. Different from the mixture of experts (MoE), our MoCE trains each expert only with semantically relevant images by using cluster-conditional gates. Thus, each downstream task can be allocated to its customized model pre-trained with data most similar to the downstream data. Experiments on a collection of 11 downstream tasks show that MoCE outperforms the vanilla MAE by 2.45% on average. It also obtains new state-of-the-art self-supervised learning results on detection and segmentation.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 56 of its 68 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

56 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Residual Mixture of Experts20 Apr
  2. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017Mask R-CNNcited by 7,259
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  13. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  14. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2016Colorful Image Colorizationcited by 1,113
  17. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  19. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  20. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  21. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  22. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  23. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  24. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  25. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  26. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  27. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  28. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  29. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  30. 2016What makes ImageNet good for transfer learning?cited by 192
  31. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  32. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  33. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  34. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  35. 2020Scalable Transfer Learning with Expert Modelscited by 48
  36. 2021SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Drivingcited by 43
  37. 2021MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Drivingcited by 41
  38. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 3; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  2. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  3. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  4. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  5. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  6. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  7. 2024Masked Image Modeling: A Surveycited by 28
  8. 2025SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensingcited by 26
  9. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  10. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  11. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Modelscited by 7
  12. 2025ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Drivingcited by 5
4 more
  1. 2024ViMoE: An Empirical Study of Designing Vision Mixture-of-Expertscited by 1
  2. 2025CoMAD: A Multiple-Teacher Self-Supervised Distillation Frameworkcited by 0
  3. 2026Learngene Search Across Multiple Datasets for Building Variable-Sized Modelscited by 0
  4. 2026Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradientscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient Masked Autoencoders with Self-Consistency
  2. 2026Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning
  3. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representations
  4. 2019Mixture-of-Experts Variational Autoencoder for Clustering and Generating from Similarity-Based Representations on Single Cell Data
  5. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  6. 2022Multimodal Masked Autoencoders Learn Transferable Representations
  7. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts
  8. 2021Lifelong Mixture of Variational Autoencoders