# When Efficiency Becomes Fragility: Exploiting Dynamic Routing Vulnerabilities in Adaptive UAV Tracking

arXiv: https://arxiv.org/abs/2608.03902 · HTML: https://mlsanity.com/paper/2608.03902/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 23 of 23)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking](https://mlsanity.com/paper/1911.07241/) (2019)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [Learning Spatially Regularized Correlation Filters for Visual Tracking](https://mlsanity.com/paper/1608.05571/) (2016)
- [Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking](https://mlsanity.com/paper/2308.06904/) (2023)
- [Cooling-Shrinking Attack: Blinding the Tracker with Imperceptible Noises](https://mlsanity.com/paper/2003.09595/) (2020)
- [IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking](https://mlsanity.com/paper/2103.14938/) (2021)
- [Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking](https://mlsanity.com/paper/2503.06625/) (2025)
- [AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild](https://mlsanity.com/paper/2602.09657/) (2026)
- [Exploring Dynamic Transformer for Efficient Object Tracking](https://mlsanity.com/paper/2403.17651/) (2024)
- [AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios](https://mlsanity.com/paper/2511.21053/) (2025)
- [TrackPGD: Efficient Adversarial Attack using Object Binary Masks against Robust Transformer Trackers](https://mlsanity.com/paper/2407.03946/) (2024)
- [CAT: Concept-level backdoor ATtacks for Concept Bottleneck Models](https://mlsanity.com/paper/2410.04823/) (2024)
- [Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking](https://mlsanity.com/paper/2412.20002/) (2024)
- [Layer-Guided UAV Tracking: Enhancing Efficiency and Occlusion Robustness](https://mlsanity.com/paper/2602.13636/) (2026)

## Abstract (arXiv)

Resource constraints on UAV platforms have driven a paradigm shift in aerial tracking, from pursuing performance toward balancing accuracy with efficiency. Adaptive Transformer Trackers, which leverage an input-dependent dynamic routing architecture, have emerged as a representative solution to this challenge. However, we reveal that behind this computation-on-demand flexibility hides a critical structural flaw: the Lipschitz singularity of computational path decisions, which has an unbounded local Lipschitz constant at discrete layer-skipping decision boundaries. This mathematical discontinuity renders adaptive tracking networks inherently unstable: tiny input perturbations can be amplified at the gating modules, causing dramatic changes in the inference topology. We formally characterize this singularity in the context of adaptive tracking architectures and, for the first time, identify it as a directly exploitable new attack surface. This insight reveals a previously overlooked and highly vulnerable topological path space attack surface. Based on this, we propose the Adversarial Path-Inversion (API) framework. API generates imperceptible perturbations to precisely manipulate the gating decisions, forcing the inference onto altered computational paths. The severe inconsistency between the original and the inverted paths dismantles the representation capability of the model. Extensive experiments on state-of-the-art adaptive trackers demonstrate that API achieves superior perturbation stealthiness, more effective attack, and faster inference speeds. This work opens a new dimension for the security analysis of dynamic tracking networks and provides a theoretical warning for constructing robust adaptive tracking architectures in the future.