# Learning Dynamic Memory Networks for Object Tracking

arXiv: https://arxiv.org/abs/1803.07268 · HTML: https://mlsanity.com/paper/1803.07268/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2018.

## Built on by (most cited first, 30 of 59)

- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [LongVLM: Efficient Long Video Understanding via Large Language Models](https://mlsanity.com/paper/2404.03384/) (2024)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024)
- [STMTrack: Template-free Visual Tracking with Space-time Memory Networks](https://mlsanity.com/paper/2104.00324/) (2021)
- [Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2107.13118/) (2021)
- [MovieChat+: Question-aware Sparse Memory for Long Video Question Answering](https://mlsanity.com/paper/2404.17176/) (2024)
- [TCTrack: Temporal Contexts for Aerial Tracking](https://mlsanity.com/paper/2203.01885/) (2022)
- [Efficient Visual Tracking with Exemplar Transformers](https://mlsanity.com/paper/2112.09686/) (2021)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [MFGNet: Dynamic Modality-Aware Filter Generation for RGB-T Tracking](https://mlsanity.com/paper/2107.10433/) (2021)
- [SMEMO: Social Memory for Trajectory Forecasting](https://mlsanity.com/paper/2203.12446/) (2022)
- [Towards Real-World Visual Tracking with Temporal Contexts](https://mlsanity.com/paper/2308.10330/) (2023)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [MV-TON: Memory-based Video Virtual Try-on network](https://mlsanity.com/paper/2108.07502/) (2021)
- [A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds](https://mlsanity.com/paper/2203.04232/) (2022)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [Memory-guided Unsupervised Image-to-image Translation](https://mlsanity.com/paper/2104.05170/) (2021)
- [Active Learning for Deep Visual Tracking](https://mlsanity.com/paper/2110.13259/) (2021)
- [Dynamic Attention guided Multi-Trajectory Analysis for Single Object Tracking](https://mlsanity.com/paper/2103.16086/) (2021)
- [Multi-Scale Memory-Based Video Deblurring](https://mlsanity.com/paper/2204.02977/) (2022)
- [Learning to Filter: Siamese Relation Network for Robust Tracking](https://mlsanity.com/paper/2104.00829/) (2021)
- [Object-Centric Multiple Object Tracking](https://mlsanity.com/paper/2309.00233/) (2023)
- [SiamMask: A Framework for Fast Online Object Tracking and Segmentation](https://mlsanity.com/paper/2207.02088/) (2022)
- [Motion-aware Memory Network for Fast Video Salient Object Detection](https://mlsanity.com/paper/2208.00946/) (2022)
- [Reading Relevant Feature from Global Representation Memory for Visual Object Tracking](https://mlsanity.com/paper/2402.14392/) (2024)
- [RTracker: Recoverable Tracking via PN Tree Structured Memory](https://mlsanity.com/paper/2403.19242/) (2024)

## Abstract (arXiv)

Template-matching methods for visual tracking have gained popularity recently due to their comparable performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. An LSTM is used as a memory controller, where the input is the search feature map and the outputs are the control signals for the reading and writing process of the memory block. As the location of the target is at first unknown in the search feature map, an attention mechanism is applied to concentrate the LSTM input on the potential target. To prevent aggressive model adaptivity, we apply gated residual template learning to control the amount of retrieved memory that is used to combine with the initial template. Unlike tracking-by-detection methods where the object's information is maintained by the weight parameters of neural networks, which requires expensive online fine-tuning to be adaptable, our tracker runs completely feed-forward and adapts to the target's appearance changes by updating the external memory. Moreover, unlike other tracking methods where the model capacity is fixed after offline training --- the capacity of our tracker can be easily enlarged as the memory requirements of a task increase, which is favorable for memorizing long-term object information. Extensive experiments on OTB and VOT demonstrates that our tracker MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed of 50 fps.