# Towards Distraction-Robust Active Visual Tracking

arXiv: https://arxiv.org/abs/2106.10110 · HTML: https://mlsanity.com/paper/2106.10110/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 29 of 29)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play](https://mlsanity.com/paper/1703.05407/) (2017)
- [Self-Supervised Policy Adaptation during Deployment](https://mlsanity.com/paper/2007.04309/) (2020)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [Deep Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1511.04143/) (2015)
- [End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning](https://mlsanity.com/paper/1808.03405/) (2018)
- [End-to-end Active Object Tracking via Reinforcement Learning](https://mlsanity.com/paper/1705.10561/) (2017)
- [Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks](https://mlsanity.com/paper/2010.13110/) (2020)
- [Inverse Reinforcement Learning in Swarm Systems](https://mlsanity.com/paper/1602.05450/) (2016)
- [Pose-Assisted Multi-Camera Collaboration for Active Object Tracking](https://mlsanity.com/paper/2001.05161/) (2020)
- [Learning to Manipulate Object Collections Using Grounded State Representations](https://mlsanity.com/paper/1909.07876/) (2019)

## Built on by (most cited first, 30 of 30)

- [ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind](https://mlsanity.com/paper/2111.09189/) (2021)
- [See and Think: Embodied Agent in Virtual Environment](https://mlsanity.com/paper/2311.15209/) (2023)
- [TrackVLA: Embodied Visual Tracking in the Wild](https://mlsanity.com/paper/2505.23189/) (2025)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy](https://mlsanity.com/paper/2407.06813/) (2024)
- [UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI](https://mlsanity.com/paper/2412.20977/) (2024)
- [Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation](https://mlsanity.com/paper/2403.08282/) (2024)
- [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024)
- [Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model](https://mlsanity.com/paper/2404.04619/) (2024)
- [TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Tracking](https://mlsanity.com/paper/2510.07134/) (2025)
- [RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking](https://mlsanity.com/paper/2304.03623/) (2023)
- [Rethinking Robustness Assessment: Adversarial Attacks on Learning-based Quadrupedal Locomotion Controllers](https://mlsanity.com/paper/2405.12424/) (2024)
- [TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification](https://mlsanity.com/paper/2209.00853/) (2022)
- [Proactive Multi-Camera Collaboration For 3D Human Pose Estimation](https://mlsanity.com/paper/2303.03767/) (2023)
- [Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL](https://mlsanity.com/paper/2207.05683/) (2022)
- [LOVON: Legged Open-Vocabulary Object Navigator](https://mlsanity.com/paper/2507.06747/) (2025)
- [Coordinate-Aligned Multi-Camera Collaboration for Active Multi-Object Tracking](https://mlsanity.com/paper/2202.10881/) (2022)
- [VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models](https://mlsanity.com/paper/2505.20718/) (2025)
- [Distracted Robot: How Visual Clutter Undermine Robotic Manipulation](https://mlsanity.com/paper/2511.22780/) (2025)
- [CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models](https://mlsanity.com/paper/2603.22846/) (2026)
- [Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision](https://mlsanity.com/paper/2207.01136/) (2022)
- [Hierarchical Instruction-aware Embodied Visual Tracking](https://mlsanity.com/paper/2505.20710/) (2025)
- [Conquering Ghosts: Relation Learning for Information Reliability Representation and End-to-End Robust Navigation](https://mlsanity.com/paper/2203.09952/) (2022)
- [Improving Robotic Manipulation Robustness via NICE Scene Surgery](https://mlsanity.com/paper/2511.22777/) (2025)
- [Instance-level Visual Active Tracking with Occlusion-Aware Planning](https://mlsanity.com/paper/2604.21453/) (2026)
- [SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.01839/) (2024)
- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [Dynamical Audio-Visual Navigation: Catching Unheard Moving Sound Sources in Unmapped 3D Environments](https://mlsanity.com/paper/2201.04279/) (2022)
- [Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization](https://mlsanity.com/paper/2607.00796/) (2026)
- [CST-WM: A Causally Structured World Model for Embodied Visual Tracking](https://mlsanity.com/paper/2609.06302/) (2026)

## Abstract (arXiv)

In active visual tracking, it is notoriously difficult when distracting objects appear, as distractors often mislead the tracker by occluding the target or bringing a confusing appearance. To address this issue, we propose a mixed cooperative-competitive multi-agent game, where a target and multiple distractors form a collaborative team to play against a tracker and make it fail to follow. Through learning in our game, diverse distracting behaviors of the distractors naturally emerge, thereby exposing the tracker's weakness, which helps enhance the distraction-robustness of the tracker. For effective learning, we then present a bunch of practical methods, including a reward function for distractors, a cross-modal teacher-student learning strategy, and a recurrent attention mechanism for the tracker. The experimental results show that our tracker performs desired distraction-robust active visual tracking and can be well generalized to unseen environments. We also show that the multi-agent game can be used to adversarially test the robustness of trackers.