mlsanityData to 7 Oct 2026

Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs

arXiv 2206.04674 ↗First posted 9 Jun 2022cs.CVcited by 58

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

To build an artificial neural network like the biological intelligence system, recent works have unified numerous tasks into a generalist model, which can process various tasks with shared parameters and do not have any task-specific modules. While generalist models achieve promising results on various benchmarks, they have performance degradation on some tasks compared with task-specialized models. In this work, we find that interference among different tasks and modalities is the main factor to this phenomenon. To mitigate such interference, we introduce the Conditional Mixture-of-Experts (Conditional MoEs) to generalist models. Routing strategies under different levels of conditions are proposed to take both the training/inference cost and generalization ability into account. By incorporating the proposed Conditional MoEs, the recently proposed generalist model Uni-Perceiver can effectively mitigate the interference across tasks and modalities, and achieves state-of-the-art results on a series of downstream tasks via prompt tuning on 1% of downstream data. Moreover, the introduction of Conditional MoEs still holds the generalization ability of generalist models to conduct zero-shot inference on new tasks, e.g., video-text retrieval and video caption. Code and pre-trained generalist models shall be released.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 81 of its 102 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

81 of 102 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2017Mask R-CNNcited by 7,259
  11. 2016Layer Normalizationcited by 5,579
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  14. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  17. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  18. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  19. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  21. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  22. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  23. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  24. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  25. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  26. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  27. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  28. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2021Going deeper with Image Transformerscited by 759
  31. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  32. 2021GPT Understands, Toocited by 715
  33. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  34. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  35. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  36. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  37. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  38. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 21; 2024: 23; 2025: 7; 2026: 3.

58 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  6. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  7. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  8. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  9. 2023ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restorationcited by 77
  10. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  11. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
  12. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
38 more
  1. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  2. 2023Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusioncited by 54
  3. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  4. 2023HumanBench: Towards General Human-centric Perception with Projector Assisted Pretrainingcited by 38
  5. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  6. 2023PDPP: Projected Diffusion for Procedure Planning in Instructional Videoscited by 36
  7. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  8. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRAcited by 36
  9. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  10. 2023Leveraging sparse and shared feature activations for disentangled representation learningcited by 32
  11. 2024All-In-One Medical Image Restoration via Task-Adaptive Routingcited by 30
  12. 2024Unifying Causal Representation Learning with the Invariance Principlecited by 30
  13. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  14. 2024Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectivescited by 27
  15. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  16. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  17. 2024GiT: Towards Generalist Vision Transformer through Universal Language Interfacecited by 23
  18. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  19. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 18
  20. 2024Graph Sparsification via Mixture of Graphscited by 17
  21. 2023Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNNcited by 15
  22. 2024Supervised Fine-tuning in turn Improves Visual Foundation Modelscited by 13
  23. 2024A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Expertscited by 13
  24. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  25. 2024UniRestorer: Universal Image Restoration via Adaptively Estimating Image Degradation at Proper Granularitycited by 12
  26. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  27. 2023Towards More Unified In-context Visual Understandingcited by 10
  28. 2024Conditional computation in neural networks: principles and research trendscited by 10
  29. 2023Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Taskscited by 9
  30. 2023DRMC: A Generalist Model with Dynamic Routing for Multi-Center PET Image Synthesiscited by 8
  31. 2024MEMoE: Enhancing Model Editing with Mixture of Experts Adaptorscited by 8
  32. 2023Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Expertscited by 7
  33. 2025Mixture of Length and Pruning Experts for Knowledge Graphs Reasoningcited by 5
  34. 2023Lightweight In-Context Tuning for Multimodal Unified Modelscited by 4
  35. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  36. 2025Towards Anytime Retrieval: A Benchmark for Anytime Person Re-Identificationcited by 4
  37. 2023Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervisioncited by 3
  38. 20233rd Place Solution for PVUW2023 VSS Track: A Large Model for Semantic Segmentation on VSPWcited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Expert Specialization in Mixture of Experts
  2. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  3. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  4. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
  5. 2025Mixture of Experts Made Intrinsically Interpretable
  6. 2026MoE Lens -- An Expert Is All You Need
  7. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  8. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts