# Visual Tracking via Dynamic Memory Networks

arXiv: https://arxiv.org/abs/1907.07613 · HTML: https://mlsanity.com/paper/1907.07613/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2019.

## Built on by (most cited first, 11 of 11)

- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [MambaEVT: Event Stream based Visual Object Tracking using State Space Model](https://mlsanity.com/paper/2408.10487/) (2024)
- [Moving Towards Centers: Re-ranking with Attention and Memory for Re-identification](https://mlsanity.com/paper/2105.01447/) (2021)
- [Weakly Supervised Video Individual CountingWeakly Supervised Video Individual Counting](https://mlsanity.com/paper/2312.05923/) (2023)
- [A Survey of Historical Learning: Learning Models with Learning History](https://mlsanity.com/paper/2303.12992/) (2023)
- [Single Object Tracking Research: A Survey](https://mlsanity.com/paper/2204.11410/) (2022)
- [Remembering What Is Important: A Factorised Multi-Head Retrieval and Auxiliary Memory Stabilisation Scheme for Human Motion Prediction](https://mlsanity.com/paper/2305.11394/) (2023)
- [Weakly Supervised Visual-Auditory Fixation Prediction with Multigranularity Perception](https://mlsanity.com/paper/2112.13697/) (2021)
- [Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking](https://mlsanity.com/paper/2506.23972/) (2025)
- [Recursive Least-Squares Estimator-Aided Online Learning for Visual Tracking](https://mlsanity.com/paper/2112.14016/) (2021)
- [DCNN: Dual Cross-current Neural Networks Realized Using An Interactive Deep Learning Discriminator for Fine-grained Objects](https://mlsanity.com/paper/2405.04093/) (2024)

## Abstract (arXiv)

Template-matching methods for visual tracking have gained popularity recently due to their good performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. The reading and writing process of the external memory is controlled by an LSTM network with the search feature map as input. A spatial attention mechanism is applied to concentrate the LSTM input on the potential target as the location of the target is at first unknown. To prevent aggressive model adaptivity, we apply gated residual template learning to control the amount of retrieved memory that is used to combine with the initial template. In order to alleviate the drift problem, we also design a "negative" memory unit that stores templates for distractors, which are used to cancel out wrong responses from the object template. To further boost the tracking performance, an auxiliary classification loss is added after the feature extractor part. Unlike tracking-by-detection methods where the object's information is maintained by the weight parameters of neural networks, which requires expensive online fine-tuning to be adaptable, our tracker runs completely feed-forward and adapts to the target's appearance changes by updating the external memory. Moreover, the capacity of our model is not determined by the network size as with other trackers --- the capacity can be easily enlarged as the memory requirements of a task increase, which is favorable for memorizing long-term object information. Extensive experiments on the OTB and VOT datasets demonstrate that our trackers perform favorably against state-of-the-art tracking methods while retaining real-time speed.