# Separable Self and Mixed Attention Transformers for Efficient Object Tracking

arXiv: https://arxiv.org/abs/2309.03979 · HTML: https://mlsanity.com/paper/2309.03979/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Separable Self-attention for Mobile Vision Transformers](https://mlsanity.com/paper/2206.02680/) (2022)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [SwinTrack: A Simple and Strong Baseline for Transformer Tracking](https://mlsanity.com/paper/2112.00995/) (2021)
- [YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video](https://mlsanity.com/paper/1702.00824/) (2017)
- [Generalized Relation Modeling for Transformer Tracking](https://mlsanity.com/paper/2303.16580/) (2023)
- [LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search](https://mlsanity.com/paper/2104.14545/) (2021)
- [DetNAS: Backbone Search for Object Detection](https://mlsanity.com/paper/1903.10979/) (2019)
- [Alpha-Refine: Boosting Tracking Performance by Precise Bounding Box Estimation](https://mlsanity.com/paper/2007.02024/) (2020)
- [HiFT: Hierarchical Feature Transformer for Aerial Tracking](https://mlsanity.com/paper/2108.00202/) (2021)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [Efficient Visual Tracking with Exemplar Transformers](https://mlsanity.com/paper/2112.09686/) (2021)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)

## Built on by (most cited first, 30 of 31)

- [Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking](https://mlsanity.com/paper/2503.06625/) (2025)
- [Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking](https://mlsanity.com/paper/2504.09228/) (2025)
- [Two-stream Beats One-stream: Asymmetric Siamese Network for Efficient Visual Tracking](https://mlsanity.com/paper/2503.00516/) (2025)
- [General Compression Framework for Efficient Transformer Object Tracking](https://mlsanity.com/paper/2409.17564/) (2024)
- [Improving Accuracy and Generalization for Efficient Visual Tracking](https://mlsanity.com/paper/2411.18855/) (2024)
- [UMDATrack: Unified Multi-Domain Adaptive Tracking Under Adverse Weather Conditions](https://mlsanity.com/paper/2507.00648/) (2025)
- [Hyperspectral Mamba for Hyperspectral Object Tracking](https://mlsanity.com/paper/2509.08265/) (2025)
- [FARTrack: Fast Autoregressive Visual Tracking with High Performance](https://mlsanity.com/paper/2602.03214/) (2026)
- [UETrack: A Unified and Efficient Framework for Single Object Tracking](https://mlsanity.com/paper/2603.01412/) (2026)
- [Learning Motion Blur Robust Vision Transformers for Real-Time UAV Tracking](https://mlsanity.com/paper/2407.05383/) (2024)
- [Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking](https://mlsanity.com/paper/2412.20002/) (2024)
- [LightFC-X: Lightweight Convolutional Tracker for RGB-X Tracking](https://mlsanity.com/paper/2502.18143/) (2025)
- [Progressive Scaling Visual Object Tracking](https://mlsanity.com/paper/2505.19990/) (2025)
- [TrackingMiM: Efficient Mamba-in-Mamba Serialization for Real-time UAV Object Tracking](https://mlsanity.com/paper/2507.01535/) (2025)
- [Multi-State Tracker: Enhancing Efficient Object Tracking via Multi-State Specialization and Interaction](https://mlsanity.com/paper/2508.11531/) (2025)
- [Dynamic Semantic-Aware Correlation Modeling for UAV Tracking](https://mlsanity.com/paper/2510.21351/) (2025)
- [Layer-Guided UAV Tracking: Enhancing Efficiency and Occlusion Robustness](https://mlsanity.com/paper/2602.13636/) (2026)
- [End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking](https://mlsanity.com/paper/2605.20569/) (2026)
- [Optimized Information Flow for Transformer Tracking](https://mlsanity.com/paper/2402.08195/) (2024)
- [Hyperspectral Adapter for Object Tracking based on Hyperspectral Video](https://mlsanity.com/paper/2503.22199/) (2025)
- [Lightweight RGB-T Tracking with Mobile Vision Transformers](https://mlsanity.com/paper/2506.19154/) (2025)
- [Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking](https://mlsanity.com/paper/2511.19928/) (2025)
- [Infrared UAV Target Tracking with Dynamic Feature Refinement and Global Contextual Attention Knowledge Distillation](https://mlsanity.com/paper/2512.04581/) (2025)
- [Exploring Reliable Spatiotemporal Dependencies for Efficient Visual Tracking](https://mlsanity.com/paper/2601.09078/) (2026)
- [CPUBone: Efficient Vision Backbone Design for Devices with Low Parallelization Capabilities](https://mlsanity.com/paper/2603.26425/) (2026)
- [Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones](https://mlsanity.com/paper/2603.26551/) (2026)
- [STATrack: A Target-Aware Fully Spiking Neural Network for Efficient UAV Tracking](https://mlsanity.com/paper/2603.27493/) (2026)
- [SFDATrack: Generalized Source-Free Domain Adaptive Tracking Under Adverse Weather Conditions](https://mlsanity.com/paper/2607.00369/) (2026)
- [MaST: Motion-aware Sparse Pipeline for Lightweight Object Tracking](https://mlsanity.com/paper/2608.24365/) (2026)
- [SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking](https://mlsanity.com/paper/2609.25503/) (2026)

## Abstract (arXiv)

The deployment of transformers for visual object tracking has shown state-of-the-art results on several benchmarks. However, the transformer-based models are under-utilized for Siamese lightweight tracking due to the computational complexity of their attention blocks. This paper proposes an efficient self and mixed attention transformer-based architecture for lightweight tracking. The proposed backbone utilizes the separable mixed attention transformers to fuse the template and search regions during feature extraction to generate superior feature encoding. Our prediction head performs global contextual modeling of the encoded features by leveraging efficient self-attention blocks for robust target state estimation. With these contributions, the proposed lightweight tracker deploys a transformer-based backbone and head module concurrently for the first time. Our ablation study testifies to the effectiveness of the proposed combination of backbone and head modules. Simulations show that our Separable Self and Mixed Attention-based Tracker, SMAT, surpasses the performance of related lightweight trackers on GOT10k, TrackingNet, LaSOT, NfS30, UAV123, and AVisT datasets, while running at 37 fps on CPU, 158 fps on GPU, and having 3.8M parameters. For example, it significantly surpasses the closely related trackers E.T.Track and MixFormerV2-S on GOT10k-test by a margin of 7.9% and 5.8%, respectively, in the AO metric. The tracker code and model is available at https://github.com/goutamyg/SMAT