# Two-stream Beats One-stream: Asymmetric Siamese Network for Efficient Visual Tracking

arXiv: https://arxiv.org/abs/2503.00516 · HTML: https://mlsanity.com/paper/2503.00516/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 58 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Pixel Difference Networks for Efficient Edge Detection](https://mlsanity.com/paper/2108.07009/) (2021)
- [Deeper and Wider Siamese Networks for Real-Time Visual Tracking](https://mlsanity.com/paper/1901.01660/) (2019)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [Generalized Relation Modeling for Transformer Tracking](https://mlsanity.com/paper/2303.16580/) (2023)
- [LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search](https://mlsanity.com/paper/2104.14545/) (2021)
- [Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance](https://mlsanity.com/paper/2403.05231/) (2024)

## Built on by (most cited first, 15 of 15)

- [SpikeTrack: A Spike-driven Framework for Efficient Visual Tracking](https://mlsanity.com/paper/2602.23963/) (2026)
- [RELO: Reinforcement Learning to Localize for Visual Object Tracking](https://mlsanity.com/paper/2605.07379/) (2026)
- [Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking](https://mlsanity.com/paper/2506.23783/) (2025)
- [FARTrack: Fast Autoregressive Visual Tracking with High Performance](https://mlsanity.com/paper/2602.03214/) (2026)
- [UETrack: A Unified and Efficient Framework for Single Object Tracking](https://mlsanity.com/paper/2603.01412/) (2026)
- [Multi-State Tracker: Enhancing Efficient Object Tracking via Multi-State Specialization and Interaction](https://mlsanity.com/paper/2508.11531/) (2025)
- [Decoupling Amplitude and Phase Attention in Frequency Domain for RGB-Event based Visual Object Tracking](https://mlsanity.com/paper/2601.01022/) (2026)
- [GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing](https://mlsanity.com/paper/2602.08550/) (2026)
- [Drift-Resilient Temporal Priors for Visual Tracking](https://mlsanity.com/paper/2604.02654/) (2026)
- [AMMNet: An Asymmetric Multi-Modal Network for Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2507.16158/) (2025)
- [Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking](https://mlsanity.com/paper/2511.19928/) (2025)
- [Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking](https://mlsanity.com/paper/2605.06112/) (2026)
- [Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking](https://mlsanity.com/paper/2606.26455/) (2026)
- [SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking](https://mlsanity.com/paper/2609.25503/) (2026)
- [LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking](https://mlsanity.com/paper/2609.33306/) (2026)

## Abstract (arXiv)

Efficient tracking has garnered attention for its ability to operate on resource-constrained platforms for real-world deployment beyond desktop GPUs. Current efficient trackers mainly follow precision-oriented trackers, adopting a one-stream framework with lightweight modules. However, blindly adhering to the one-stream paradigm may not be optimal, as incorporating template computation in every frame leads to redundancy, and pervasive semantic interaction between template and search region places stress on edge devices. In this work, we propose a novel asymmetric Siamese tracker named AsymTrack for efficient tracking. AsymTrack disentangles template and search streams into separate branches, with template computing only once during initialization to generate modulation signals. Building on this architecture, we devise an efficient template modulation mechanism to unidirectional inject crucial cues into the search features, and design an object perception enhancement module that integrates abstract semantics and local details to overcome the limited representation in lightweight tracker. Extensive experiments demonstrate that AsymTrack offers superior speed-precision trade-offs across different platforms compared to the current state-of-the-arts. For instance, AsymTrack-T achieves 60.8% AUC on LaSOT and 224/81/84 FPS on GPU/CPU/AGX, surpassing HiT-Tiny by 6.0% AUC with higher speeds. The code is available at https://github.com/jiawen-zhu/AsymTrack.