# MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model

arXiv: https://arxiv.org/abs/2405.00293 · HTML: https://mlsanity.com/paper/2405.00293/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts.
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 1 May 2024.

## Placed in

- Mixture-of-experts / [Mixture of LoRA experts](https://mlsanity.com/moe/mixture-of-lora-experts/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding](https://mlsanity.com/paper/2012.02951/) (2020)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [Semantic Segmentation of Underwater Imagery: Dataset and Benchmark](https://mlsanity.com/paper/2004.01241/) (2020)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot](https://mlsanity.com/paper/2306.16623/) (2023)
- [Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy](https://mlsanity.com/paper/2011.08065/) (2020)
- [CryoNuSeg: A Dataset for Nuclei Instance Segmentation of Cryosectioned H&E-Stained Histological Images](https://mlsanity.com/paper/2101.00442/) (2021)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm](https://mlsanity.com/paper/2104.08682/) (2021)

## Built on by (most cited first, 3 of 3)

- [Swin-TUNA : A Novel PEFT Approach for Accurate Food Image Segmentation](https://mlsanity.com/paper/2507.17347/) (2025)
- [Are Vision Foundation Models Foundational for Electron Microscopy Image Segmentation?](https://mlsanity.com/paper/2602.08505/) (2026)
- [MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting](https://mlsanity.com/paper/2511.12400/) (2025)

## Abstract (arXiv)

The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.