# SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking

arXiv: https://arxiv.org/abs/2503.18338 · HTML: https://mlsanity.com/paper/2503.18338/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 52 of its 71 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [AiATrack: Attention in Attention for Transformer Visual Tracking](https://mlsanity.com/paper/2207.09603/) (2022)
- [ODTrack: Online Dense Temporal Token Learning for Visual Tracking](https://mlsanity.com/paper/2401.01686/) (2024)

## Built on by (most cited first, 24 of 24)

- [Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization](https://mlsanity.com/paper/2506.17252/) (2025)
- [Heterogeneous Agent Collaborative Reinforcement Learning](https://mlsanity.com/paper/2603.02604/) (2026)
- [Real-Time Aligned Reward Model beyond Semantics](https://mlsanity.com/paper/2601.22664/) (2026)
- [SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker](https://mlsanity.com/paper/2604.12502/) (2026)
- [RELO: Reinforcement Learning to Localize for Visual Object Tracking](https://mlsanity.com/paper/2605.07379/) (2026)
- [UETrack: A Unified and Efficient Framework for Single Object Tracking](https://mlsanity.com/paper/2603.01412/) (2026)
- [Tracking and Segmenting Anything in Any Modality](https://mlsanity.com/paper/2511.19475/) (2025)
- [SurgAtt-Tracker: Online Surgical Attention Tracking via Temporal Proposal Reranking and Motion-Aware Refinement](https://mlsanity.com/paper/2602.20636/) (2026)
- [Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modality](https://mlsanity.com/paper/2603.14452/) (2026)
- [Drift-Resilient Temporal Priors for Visual Tracking](https://mlsanity.com/paper/2604.02654/) (2026)
- [Unified Multimodal Visual Tracking with Dual Mixture-of-Experts](https://mlsanity.com/paper/2605.03716/) (2026)
- [Boosting Self-Supervised Tracking with Contextual Prompts and Noise Learning](https://mlsanity.com/paper/2605.06092/) (2026)
- [An Efficient Token Compression Framework for Visual Object Tracking](https://mlsanity.com/paper/2605.08329/) (2026)
- [SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking](https://mlsanity.com/paper/2606.24449/) (2026)
- [Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices](https://mlsanity.com/paper/2507.14959/) (2025)
- [What really matters for person re-identification? A Mixture-of-Experts Framework for Semantic Attribute Importance](https://mlsanity.com/paper/2512.08697/) (2025)
- [Exploring Reliable Spatiotemporal Dependencies for Efficient Visual Tracking](https://mlsanity.com/paper/2601.09078/) (2026)
- [Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking](https://mlsanity.com/paper/2603.13719/) (2026)
- [SpecTrack: Spectral Prompt Guided Adaptive Experts for Multispectral Object Tracking](https://mlsanity.com/paper/2607.05988/) (2026)
- [Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Tracking](https://mlsanity.com/paper/2608.00847/) (2026)
- [SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control](https://mlsanity.com/paper/2608.07876/) (2026)
- [FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking](https://mlsanity.com/paper/2608.09369/) (2026)
- [Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioning](https://mlsanity.com/paper/2609.01136/) (2026)
- [Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking](https://mlsanity.com/paper/2610.05707/) (2026)

## Abstract (arXiv)

Most state-of-the-art trackers adopt one-stream paradigm, using a single Vision Transformer for joint feature extraction and relation modeling of template and search region images. However, relation modeling between different image patches exhibits significant variations. For instance, background regions dominated by target-irrelevant information require reduced attention allocation, while foreground, particularly boundary areas, need to be be emphasized. A single model may not effectively handle all kinds of relation modeling simultaneously. In this paper, we propose a novel tracker called SPMTrack based on mixture-of-experts tailored for visual tracking task (TMoE), combining the capability of multiple experts to handle diverse relation modeling more flexibly. Benefiting from TMoE, we extend relation modeling from image pairs to spatio-temporal context, further improving tracking accuracy with minimal increase in model parameters. Moreover, we employ TMoE as a parameter-efficient fine-tuning method, substantially reducing trainable parameters, which enables us to train SPMTrack of varying scales efficiently and preserve the generalization ability of pretrained models to achieve superior performance. We conduct experiments on seven datasets, and experimental results demonstrate that our method significantly outperforms current state-of-the-art trackers. The source code is available at https://github.com/WenRuiCai/SPMTrack.