mlsanityData to 7 Oct 2026

MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model

arXiv 2405.00293 ↗First posted 1 May 2024cs.CVcited by 3

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 35 of its 50 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

35 of 50 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Segment Anythingcited by 8,158
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
23 more
  1. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  2. 2023Segment Anything in High Qualitycited by 373
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  5. 2023Track Anything: Segment Anything Meets Videoscited by 230
  6. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
  7. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  8. 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
  9. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  10. 2020Semantic Segmentation of Underwater Imagery: Dataset and Benchmarkcited by 95
  11. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  12. 2019PST900: RGB-Thermal Calibration, Dataset and Segmentation Networkcited by 85
  13. 2023The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shotcited by 83
  14. 2020Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopycited by 54
  15. 2021CryoNuSeg: A Dataset for Nuclei Instance Segmentation of Cryosectioned H&E-Stained Histological Imagescited by 50
  16. 2021A Large-Scale Benchmark for Food Image Segmentationcited by 45
  17. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  18. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  19. 2022Detailed Annotations of Chest X-Rays via CT Projection for Report Understandingcited by 24
  20. 2023What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentationcited by 24
  21. 2022Semantic Segmentation in Art Paintingscited by 23
  22. 2024BiMediX: Bilingual Medical Mixture of Experts LLMcited by 21
  23. 2021ATLANTIS: A Benchmark for Semantic Segmentation of Waterbody Imagescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Swin-TUNA : A Novel PEFT Approach for Accurate Food Image Segmentationcited by 1
  2. 2026Are Vision Foundation Models Foundational for Electron Microscopy Image Segmentation?cited by 1
  3. 2025MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweightingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  2. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
  3. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
  4. 2025MoFE: Mixture of Frozen Experts Architecture
  5. 2024Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models
  6. 2023Revisiting Single-gated Mixtures of Experts
  7. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
  8. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model