mlsanityData to 7 Oct 2026

PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization

arXiv 2606.03444 ↗First posted 2 Jun 2026cs.CV cs.AIcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Unifying the complementary strengths of diverse Vision Foundation Models (VFMs) into a single efficient model is highly desirable but challenged by the negative transfer inherent in monolithic distillation. To address these feature conflicts, we introduce PRISM, a novel dual-stream Mixture-of-Experts (MoE) framework that synergizes VFMs via modular specialization. We propose a two-stage paradigm: (1) expertise deconstruction, where a teacher-conditional router guides experts to specialize in distinct representational subspaces to mitigate interference, followed by (2) dynamic recomposition, where the router learns to assemble these experts into tailored computational pathways for downstream tasks. Experiments on PASCAL-Context and NYUD-v2 show that PRISM establishes a new state of the art, validating that sparse, emergent specialization is a scalable approach for integrating diverse visual knowledge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 48 of its 66 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

48 of 66 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  8. 2017A Survey on Multi-Task Learningcited by 838
  9. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  11. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  12. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
36 more
  1. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  2. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  3. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  4. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  5. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  6. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  7. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  8. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  9. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  10. 2022Transformer Tracking with Cyclic Shifting Window Attentioncited by 75
  11. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  12. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  13. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  14. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  15. 2023Compact Transformer Tracker with Correlative Masked Modelingcited by 50
  16. 2021Multi-task Learning with Attention for End-to-end Autonomous Drivingcited by 49
  17. 2020Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interferencecited by 47
  18. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  19. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  20. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
  21. 2024Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Erascited by 26
  22. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 24
  23. 2022Universal Representations: A Unified Look at Multiple Task and Domain Learningcited by 23
  24. 2023InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understandingcited by 22
  25. 2024Autogenic Language Embedding for Coherent Point Trackingcited by 17
  26. 2024All-in-One: Transferring Vision Foundation Models into Stereo Matchingcited by 16
  27. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  28. 2025MVP: Winning Solution to SMP Challenge 2025 Video Trackcited by 14
  29. 2023TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understandingcited by 12
  30. 2023BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-task Dense Predictionscited by 12
  31. 2023Efficient Computation Sharing for Multi-Task Visual Scene Understandingcited by 8
  32. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  33. 2023Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulationcited by 5
  34. 2026Seeing Further and Wider: Joint Spatio-Temporal Enlargement for Micro-Video Popularity Predictioncited by 5
  35. 2026CurEvo: Curriculum-Guided Self-Evolution for Video Understandingcited by 3
  36. 20253D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understandingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  2. 2025Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
  3. 2024ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts
  4. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
  5. 2026Multi-Domain Learning with Global Expert Mapping
  6. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
  7. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models
  8. 2023Prismer: A Vision-Language Model with Multi-Task Experts