# Video Object Detection with an Aligned Spatial-Temporal Memory

arXiv: https://arxiv.org/abs/1712.06317 · HTML: https://mlsanity.com/paper/1712.06317/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2017.

## Built on by (most cited first, 30 of 42)

- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection](https://mlsanity.com/paper/2205.05979/) (2022)
- [Temporal RoI Align for Video Object Recognition](https://mlsanity.com/paper/2109.03495/) (2021)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)
- [Monocular 3D Object Detection with Depth from Motion](https://mlsanity.com/paper/2207.12988/) (2022)
- [TF-Blender: Temporal Feature Blender for Video Object Detection](https://mlsanity.com/paper/2108.05821/) (2021)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Object Permanence Emerges in a Random Walk along Memory](https://mlsanity.com/paper/2204.01784/) (2022)
- [A Guide to Image and Video based Small Object Detection using Deep Learning : Case Study of Maritime Surveillance](https://mlsanity.com/paper/2207.12926/) (2022)
- [INT: Towards Infinite-frames 3D Detection with An Efficient Framework](https://mlsanity.com/paper/2209.15215/) (2022)
- [InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation](https://mlsanity.com/paper/2301.01882/) (2023)
- [DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving](https://mlsanity.com/paper/2303.17144/) (2023)
- [QueryProp: Object Query Propagation for High-Performance Video Object Detection](https://mlsanity.com/paper/2207.10959/) (2022)
- [LongShortNet: Exploring Temporal and Semantic Features Fusion in Streaming Perception](https://mlsanity.com/paper/2210.15518/) (2022)
- [MaCLR: Motion-aware Contrastive Learning of Representations for Videos](https://mlsanity.com/paper/2106.09703/) (2021)
- [A recurrent CNN for online object detection on raw radar frames](https://mlsanity.com/paper/2212.11172/) (2022)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Few-Shot Video Object Detection](https://mlsanity.com/paper/2104.14805/) (2021)
- [Objects do not disappear: Video object detection by single-frame object location anticipation](https://mlsanity.com/paper/2308.04770/) (2023)
- [Deep Learning for Weakly-Supervised Object Detection and Object Localization: A Survey](https://mlsanity.com/paper/2105.12694/) (2021)
- [BoxMask: Revisiting Bounding Box Supervision for Video Object Detection](https://mlsanity.com/paper/2210.06008/) (2022)
- [Lightweight Multi-Frame Integration for Robust YOLO Object Detection in Videos](https://mlsanity.com/paper/2506.20550/) (2025)
- [Identity-Consistent Aggregation for Video Object Detection](https://mlsanity.com/paper/2308.07737/) (2023)
- [Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception](https://mlsanity.com/paper/2409.06584/) (2024)
- [Learning to Track Object Position through Occlusion](https://mlsanity.com/paper/2106.10766/) (2021)
- [FFAVOD: Feature Fusion Architecture for Video Object Detection](https://mlsanity.com/paper/2109.07298/) (2021)
- [Explore Spatio-temporal Aggregation for Insubstantial Object Detection: Benchmark Dataset and Baseline](https://mlsanity.com/paper/2206.11459/) (2022)

## Abstract (arXiv)

We introduce Spatial-Temporal Memory Networks for video object detection. At its core, a novel Spatial-Temporal Memory module (STMM) serves as the recurrent computation unit to model long-term temporal appearance and motion dynamics. The STMM's design enables full integration of pretrained backbone CNN weights, which we find to be critical for accurate detection. Furthermore, in order to tackle object motion in videos, we propose a novel MatchTrans module to align the spatial-temporal memory from frame to frame. Our method produces state-of-the-art results on the benchmark ImageNet VID dataset, and our ablative studies clearly demonstrate the contribution of our different design choices. We release our code and models at http://fanyix.cs.ucdavis.edu/project/stmn/project.html.